[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:32:47 dist-test-slave-svn8 krb5kdc[25334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:32:47 dist-test-slave-svn8 krb5kdc[25334](info): set up 2 sockets
Mar 19 16:32:47 dist-test-slave-svn8 krb5kdc[25334](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260319 16:32:49.730377 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:32:49 dist-test-slave-svn8 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:32:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:32:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:49.868328 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33619
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:33619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:49.943168 25351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:49.943356 25351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:49.943378 25351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:49.944895 25351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:32:49.944949 25351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:49.944963 25351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:49.944974 25351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:32:49.944985 25351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:32:49.946990 25351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:33619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:49.947211 25351 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:49.947381 25351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:49.949980 25357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:49.949985 25356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:49.949986 25359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:49.950004 25351 server_base.cc:1060] running on GCE node
I20260319 16:32:49.950340 25351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:49.950613 25351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:49.951771 25351 hybrid_clock.cc:648] HybridClock initialized: now 1773937969951748 us; error 34 us; skew 500 ppm
Mar 19 16:32:49 dist-test-slave-svn8 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937969, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:49.954241 25351 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:32:49.954777 25351 webserver.cc:492] Webserver started at http://127.24.187.254:44207/ using document root <none> and password file <none>
I20260319 16:32:49.954945 25351 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:49.954981 25351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:49.955080 25351 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:49.956018 25351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7"
format_stamp: "Formatted at 2026-03-19 16:32:49 on dist-test-slave-svn8"
server_key: "e6ae3d377efbfe8e42ae879cec45dcfc"
server_key_iv: "f84ca97120704c09f378bd12c30ba98c"
server_key_version: "encryptionkey@0"
I20260319 16:32:49.956398 25351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7"
format_stamp: "Formatted at 2026-03-19 16:32:49 on dist-test-slave-svn8"
server_key: "e6ae3d377efbfe8e42ae879cec45dcfc"
server_key_iv: "f84ca97120704c09f378bd12c30ba98c"
server_key_version: "encryptionkey@0"
I20260319 16:32:49.957662 25351 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:32:49.958459 25366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:49.958694 25351 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:32:49.958765 25351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7"
format_stamp: "Formatted at 2026-03-19 16:32:49 on dist-test-slave-svn8"
server_key: "e6ae3d377efbfe8e42ae879cec45dcfc"
server_key_iv: "f84ca97120704c09f378bd12c30ba98c"
server_key_version: "encryptionkey@0"
I20260319 16:32:49.958828 25351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:49.976220 25351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:49.982488 25351 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:49.982632 25351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:49.986598 25351 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:33619
I20260319 16:32:49.986625 25418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:33619 every 8 connection(s)
I20260319 16:32:49.986935 25351 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:32:49.987517 25419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:49.989907 25419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7: Bootstrap starting.
I20260319 16:32:49.990453 25419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:49.990670 25419 log.cc:826] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:49.991321 25419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7: No bootstrap required, opened a new log
I20260319 16:32:49.992520 25419 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33619 } }
I20260319 16:32:49.992637 25419 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:49.992660 25419 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7c93ce07ee4f96a22ee45424b6c2a7, State: Initialized, Role: FOLLOWER
I20260319 16:32:49.992769 25419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33619 } }
I20260319 16:32:49.992828 25419 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:32:49.992857 25419 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:32:49.992901 25419 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:49.993017 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 25351
I20260319 16:32:49.993108 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:32:49.993497 25419 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33619 } }
I20260319 16:32:49.993599 25419 leader_election.cc:304] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7c93ce07ee4f96a22ee45424b6c2a7; no voters:
I20260319 16:32:49.993700 25327 external_mini_cluster.cc:1468] Setting key cc84171d54d1d4a46884adb6c66ff6d6
I20260319 16:32:49.993772 25419 leader_election.cc:290] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:32:49.993803 25422 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:32:49.993942 25422 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [term 1 LEADER]: Becoming Leader. State: Replica: 2d7c93ce07ee4f96a22ee45424b6c2a7, State: Running, Role: LEADER
I20260319 16:32:49.993999 25419 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:32:49.994084 25422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33619 } }
I20260319 16:32:49.994637 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7c93ce07ee4f96a22ee45424b6c2a7. Latest consensus state: current_term: 1 leader_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33619 } } }
I20260319 16:32:49.994820 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [sys.catalog]: This master's current role is: LEADER
I20260319 16:32:49.995018 25424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c93ce07ee4f96a22ee45424b6c2a7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33619 } } }
I20260319 16:32:49.995143 25424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7 [sys.catalog]: This master's current role is: LEADER
I20260319 16:32:49.995261 25433 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:32:49.995760 25433 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:32:49.997195 25433 catalog_manager.cc:1357] Generated new cluster ID: 3d14ba84776b4740878bdaa6228f1c26
I20260319 16:32:49.997242 25433 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:50.008265 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:49.994390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.24.187.254:33619)
0319 16:32:49.994537 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:49.994548 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:49.994784 (+ 236us) server_negotiation.cc:408] Connection header received
0319 16:32:49.996639 (+ 1855us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:49.996646 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:49.996699 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:49.996808 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:49.998337 (+ 1529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:49.999576 (+ 1239us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.000355 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.000624 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.004047 (+ 3423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:50.004053 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:50.004054 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:50.004076 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:50.006670 (+ 2594us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.007118 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.007121 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.007121 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.007176 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.007367 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.007367 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.007368 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.007587 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:50.007713 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:50.007962 (+ 249us) server_negotiation.cc:300] Negotiation successful
0319 16:32:50.008080 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260319 16:32:50.018677 25433 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:32:50.019436 25433 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:32:50.023948 25433 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2d7c93ce07ee4f96a22ee45424b6c2a7: Generated new TSK 0
I20260319 16:32:50.024114 25433 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:50.082372 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33619
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:50.161736 25447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:50.161974 25447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:50.161999 25447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:50.163547 25447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:50.163596 25447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:50.163622 25447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:32:50.165256 25447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:33619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:50.165450 25447 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:50.165587 25447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:50.168313 25455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:50.168449 25453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:50.168567 25447 server_base.cc:1060] running on GCE node
W20260319 16:32:50.168627 25452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:50.168777 25447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:50.168962 25447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:50.170115 25447 hybrid_clock.cc:648] HybridClock initialized: now 1773937970170097 us; error 24 us; skew 500 ppm
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:50.172415 25447 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:32:50.172933 25447 webserver.cc:492] Webserver started at http://127.24.187.193:39719/ using document root <none> and password file <none>
I20260319 16:32:50.173094 25447 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:50.173127 25447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:50.173237 25447 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:50.174196 25447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "fd22e3c0a9f34322adf9453e6bee4c86"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "56fea12a00f2c1b47c759bf644ed0143"
server_key_iv: "171b3bc4b8ded3b0f375eee6a4731bac"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.174484 25447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "fd22e3c0a9f34322adf9453e6bee4c86"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "56fea12a00f2c1b47c759bf644ed0143"
server_key_iv: "171b3bc4b8ded3b0f375eee6a4731bac"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.175595 25447 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:32:50.176303 25462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.176517 25447 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:32:50.176577 25447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "fd22e3c0a9f34322adf9453e6bee4c86"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "56fea12a00f2c1b47c759bf644ed0143"
server_key_iv: "171b3bc4b8ded3b0f375eee6a4731bac"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.176636 25447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:50.190140 25447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:50.192095 25447 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:50.192231 25447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:50.192466 25447 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:32:50.192812 25447 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:50.192847 25447 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.192875 25447 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:50.192891 25447 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.200724 25447 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:42341
I20260319 16:32:50.201027 25575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:42341 every 8 connection(s)
I20260319 16:32:50.201146 25447 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:50.208086 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 25447
I20260319 16:32:50.208155 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:32:50.208295 25327 external_mini_cluster.cc:1468] Setting key 7cd48b002ad8eb9e565fb1dc6ec72b69
I20260319 16:32:50.211028 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.202487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:60311 (local address 127.24.187.254:33619)
0319 16:32:50.202560 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:50.202562 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:50.202632 (+ 70us) server_negotiation.cc:408] Connection header received
0319 16:32:50.203181 (+ 549us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:50.203183 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:50.203193 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:50.203253 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:50.204200 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.204743 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.205293 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.205400 (+ 107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.207821 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:50.207826 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:50.207827 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:50.207834 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:50.209624 (+ 1790us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.210086 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.210087 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.210088 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.210117 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.210359 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.210360 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.210360 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.210476 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:50.210549 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:50.210823 (+ 274us) server_negotiation.cc:300] Negotiation successful
0319 16:32:50.210934 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260319 16:32:50.211139 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.202389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:33619 (local address 127.24.187.193:60311)
0319 16:32:50.202525 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:50.202554 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:50.203086 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:50.203496 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:50.203501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:50.203824 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:50.204099 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:50.204103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.204849 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.204850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:50.205226 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:50.205228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.205326 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.206234 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:50.206250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:50.207724 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:50.209773 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:50.209781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:50.209784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:50.210012 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:50.210215 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:50.210216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:50.210217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:50.210296 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:50.210536 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:50.210540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:50.210658 (+ 118us) client_negotiation.cc:769] Sending connection context
0319 16:32:50.210775 (+ 117us) client_negotiation.cc:240] Negotiation successful
0319 16:32:50.210938 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260319 16:32:50.211730 25576 heartbeater.cc:344] Connected to a master server at 127.24.187.254:33619
I20260319 16:32:50.211839 25576 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:50.212031 25576 heartbeater.cc:507] Master 127.24.187.254:33619 requested a full tablet report, sending...
I20260319 16:32:50.212581 25383 ts_manager.cc:194] Registered new tserver with Master: fd22e3c0a9f34322adf9453e6bee4c86 (127.24.187.193:42341)
I20260319 16:32:50.213229 25383 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:60311
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:50.257606 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33619
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:50.334193 25583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:50.334362 25583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:50.334403 25583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:50.335845 25583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:50.335886 25583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:50.335920 25583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:32:50.337507 25583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:33619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:50.337740 25583 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:50.337903 25583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:50.340487 25589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:50.340710 25588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:50.340749 25591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:50.340822 25583 server_base.cc:1060] running on GCE node
I20260319 16:32:50.340952 25583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:50.341159 25583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:50.342290 25583 hybrid_clock.cc:648] HybridClock initialized: now 1773937970342276 us; error 27 us; skew 500 ppm
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:50.344695 25583 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:32:50.345206 25583 webserver.cc:492] Webserver started at http://127.24.187.194:35569/ using document root <none> and password file <none>
I20260319 16:32:50.345382 25583 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:50.345417 25583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:50.345505 25583 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:50.346490 25583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "eb491a4f7d754a77ae24ee137a6a0490"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "82a2ea4d84c8caa4868058c74015d04c"
server_key_iv: "3e9c4f1b9a4061bd4afda8bf5a4a1ce9"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.346779 25583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "eb491a4f7d754a77ae24ee137a6a0490"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "82a2ea4d84c8caa4868058c74015d04c"
server_key_iv: "3e9c4f1b9a4061bd4afda8bf5a4a1ce9"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.347903 25583 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:32:50.348665 25598 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.348870 25583 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:32:50.348953 25583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "eb491a4f7d754a77ae24ee137a6a0490"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "82a2ea4d84c8caa4868058c74015d04c"
server_key_iv: "3e9c4f1b9a4061bd4afda8bf5a4a1ce9"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.349014 25583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:50.360821 25583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:50.362875 25583 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:50.363008 25583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:50.363216 25583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:32:50.363560 25583 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:50.363595 25583 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.363623 25583 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:50.363648 25583 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.369203 25583 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:37177
I20260319 16:32:50.369257 25711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:37177 every 8 connection(s)
I20260319 16:32:50.369540 25583 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:32:50.372040 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 25583
I20260319 16:32:50.372116 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:32:50.372249 25327 external_mini_cluster.cc:1468] Setting key a888c067aee2e08eacaa72ed6a3ffa66
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:50.378731 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.370371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:55905 (local address 127.24.187.254:33619)
0319 16:32:50.370516 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:50.370519 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:50.370646 (+ 127us) server_negotiation.cc:408] Connection header received
0319 16:32:50.371147 (+ 501us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:50.371148 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:50.371161 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:50.371213 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:50.371946 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.372471 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.372978 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.373133 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.375949 (+ 2816us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:50.375954 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:50.375954 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:50.375961 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:50.377394 (+ 1433us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.377824 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.377825 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.377826 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.377850 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.378009 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.378010 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.378010 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.378169 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:50.378258 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:50.378528 (+ 270us) server_negotiation.cc:300] Negotiation successful
0319 16:32:50.378634 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260319 16:32:50.378837 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.370261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:33619 (local address 127.24.187.194:55905)
0319 16:32:50.370543 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:50.370573 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:50.371062 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:50.371266 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:50.371270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:50.371575 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:50.371855 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:50.371859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.372546 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.372547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:50.372910 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:50.372912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.373024 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.374152 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:50.374167 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:50.375843 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:50.377491 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:50.377497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:50.377499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:50.377760 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:50.377899 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:50.377899 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:50.377900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:50.377961 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:50.378265 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:50.378270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:50.378402 (+ 132us) client_negotiation.cc:769] Sending connection context
0319 16:32:50.378500 (+ 98us) client_negotiation.cc:240] Negotiation successful
0319 16:32:50.378642 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
I20260319 16:32:50.379253 25712 heartbeater.cc:344] Connected to a master server at 127.24.187.254:33619
I20260319 16:32:50.379338 25712 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:50.379528 25712 heartbeater.cc:507] Master 127.24.187.254:33619 requested a full tablet report, sending...
I20260319 16:32:50.379894 25383 ts_manager.cc:194] Registered new tserver with Master: eb491a4f7d754a77ae24ee137a6a0490 (127.24.187.194:37177)
I20260319 16:32:50.380273 25383 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:55905
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:50.425530 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33619
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:50.507072 25719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:50.507279 25719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:50.507305 25719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:50.509078 25719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:50.509128 25719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:50.509156 25719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:32:50.511000 25719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:33619
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:50.511212 25719 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:50.511386 25719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:50.513957 25725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:50.513906 25724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:50.513890 25727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:50.513940 25719 server_base.cc:1060] running on GCE node
I20260319 16:32:50.514279 25719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:50.514480 25719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:50.515652 25719 hybrid_clock.cc:648] HybridClock initialized: now 1773937970515631 us; error 35 us; skew 500 ppm
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:50.517936 25719 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:32:50.518420 25719 webserver.cc:492] Webserver started at http://127.24.187.195:34501/ using document root <none> and password file <none>
I20260319 16:32:50.518577 25719 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:50.518611 25719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:50.518697 25719 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:50.519609 25719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "bdee983d56e6462da712fd330b8dfe0d"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "33657d381d81954e3f365e076e972dee"
server_key_iv: "253bcf005d57cea2212f473b17831e9e"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.519892 25719 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "bdee983d56e6462da712fd330b8dfe0d"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "33657d381d81954e3f365e076e972dee"
server_key_iv: "253bcf005d57cea2212f473b17831e9e"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.521009 25719 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:32:50.521701 25734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.521885 25719 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:32:50.521937 25719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "bdee983d56e6462da712fd330b8dfe0d"
format_stamp: "Formatted at 2026-03-19 16:32:50 on dist-test-slave-svn8"
server_key: "33657d381d81954e3f365e076e972dee"
server_key_iv: "253bcf005d57cea2212f473b17831e9e"
server_key_version: "encryptionkey@0"
I20260319 16:32:50.521996 25719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:50.557350 25719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:50.559031 25719 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:50.559175 25719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:50.559396 25719 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:32:50.559733 25719 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:50.559764 25719 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.559787 25719 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:50.559809 25719 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:50.565342 25719 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:36657
I20260319 16:32:50.565486 25847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:36657 every 8 connection(s)
I20260319 16:32:50.565680 25719 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:50.571061 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 25719
I20260319 16:32:50.571149 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:32:50.571303 25327 external_mini_cluster.cc:1468] Setting key 194f571237abbf64151c742d44bd07c4
I20260319 16:32:50.573537 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.566350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:54669 (local address 127.24.187.254:33619)
0319 16:32:50.566408 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:50.566409 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:50.566565 (+ 156us) server_negotiation.cc:408] Connection header received
0319 16:32:50.567051 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:50.567052 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:50.567061 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:50.567112 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:50.567887 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.568370 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.568861 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.568966 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.571266 (+ 2300us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:50.571272 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:50.571272 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:50.571280 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:50.572576 (+ 1296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.572894 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.572894 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.572895 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.572911 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.573052 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.573053 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.573053 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.573135 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:50.573175 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:50.573411 (+ 236us) server_negotiation.cc:300] Negotiation successful
0319 16:32:50.573475 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:32:50.573693 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.566310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:33619 (local address 127.24.187.195:54669)
0319 16:32:50.566460 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:50.566491 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:50.566982 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:50.567159 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:50.567162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:50.567506 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:50.567796 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:50.567801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.568434 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.568434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:50.568807 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:50.568808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.568893 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.569679 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:50.569731 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:50.571129 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:50.572642 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:50.572646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:50.572648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:50.572838 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:50.572963 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:50.572964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:50.572964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:50.573014 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:50.573174 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:50.573176 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:50.573287 (+ 111us) client_negotiation.cc:769] Sending connection context
0319 16:32:50.573368 (+ 81us) client_negotiation.cc:240] Negotiation successful
0319 16:32:50.573512 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260319 16:32:50.574188 25848 heartbeater.cc:344] Connected to a master server at 127.24.187.254:33619
I20260319 16:32:50.574265 25848 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:50.574414 25848 heartbeater.cc:507] Master 127.24.187.254:33619 requested a full tablet report, sending...
I20260319 16:32:50.574743 25383 ts_manager.cc:194] Registered new tserver with Master: bdee983d56e6462da712fd330b8dfe0d (127.24.187.195:36657)
I20260319 16:32:50.575085 25383 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:54669
I20260319 16:32:50.583504 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:32:50.605186 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.598290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.24.187.193:42341)
0319 16:32:50.598447 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:50.598453 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:50.598461 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:32:50.598486 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:50.598488 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:50.598500 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:50.598559 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:50.599256 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.599996 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.600529 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.600733 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.602690 (+ 1957us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:50.602719 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:50.602719 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:50.602726 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:50.604202 (+ 1476us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.604519 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.604522 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.604523 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.604555 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.604704 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.604704 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.604705 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.604797 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:50.604919 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:50.604995 (+ 76us) server_negotiation.cc:300] Negotiation successful
0319 16:32:50.605078 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
W20260319 16:32:50.605587 25472 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39622
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:32:50.626969 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:50.620322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.24.187.193:42341)
0319 16:32:50.620378 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:50.620380 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:50.620468 (+ 88us) server_negotiation.cc:408] Connection header received
0319 16:32:50.620519 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:50.620520 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:50.620531 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:50.620582 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:50.621178 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.621671 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:50.622254 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:50.622387 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:50.624475 (+ 2088us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:50.624480 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:50.624480 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:50.624486 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:50.625948 (+ 1462us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.626331 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.626332 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.626332 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.626358 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:50.626513 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:50.626513 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:50.626513 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:50.626584 (+ 71us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:50.626657 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:50.626831 (+ 174us) server_negotiation.cc:300] Negotiation successful
0319 16:32:50.626899 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:32:50.627470 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 25447
I20260319 16:32:50.632586 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 25583
I20260319 16:32:50.638090 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 25719
I20260319 16:32:50.641943 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 25351
2026-03-19T16:32:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2757 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25862](info): set up 2 sockets
Mar 19 16:32:50 dist-test-slave-svn8 krb5kdc[25862](info): commencing operation
krb5kdc: starting...
W20260319 16:32:52.490291 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.829s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:32:52 dist-test-slave-svn8 krb5kdc[25862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:32:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:32:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:52.623927 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38087
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:38087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:52.696511 25878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:52.696694 25878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:52.696724 25878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:52.698220 25878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:32:52.698261 25878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:52.698279 25878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:52.698297 25878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:32:52.698309 25878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:32:52.699916 25878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:38087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:52.700120 25878 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:52.700278 25878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:32:52.702673 25878 server_base.cc:1060] running on GCE node
W20260319 16:32:52.702509 25883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:52.702517 25886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:52.702512 25884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:52.703007 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:52.703202 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:52.704344 25878 hybrid_clock.cc:648] HybridClock initialized: now 1773937972704309 us; error 48 us; skew 500 ppm
Mar 19 16:32:52 dist-test-slave-svn8 krb5kdc[25862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937972, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:52.706589 25878 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:32:52.707067 25878 webserver.cc:492] Webserver started at http://127.24.187.254:44007/ using document root <none> and password file <none>
I20260319 16:32:52.707207 25878 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:52.707237 25878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:52.707309 25878 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:52.708163 25878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "2c44f78c107e49acaaa47f6c60392144"
format_stamp: "Formatted at 2026-03-19 16:32:52 on dist-test-slave-svn8"
server_key: "0eb5dcbb50e4cbc440a4732c04c83131"
server_key_iv: "1caafddc49987170e2f354b7485a9f47"
server_key_version: "encryptionkey@0"
I20260319 16:32:52.708425 25878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "2c44f78c107e49acaaa47f6c60392144"
format_stamp: "Formatted at 2026-03-19 16:32:52 on dist-test-slave-svn8"
server_key: "0eb5dcbb50e4cbc440a4732c04c83131"
server_key_iv: "1caafddc49987170e2f354b7485a9f47"
server_key_version: "encryptionkey@0"
I20260319 16:32:52.709467 25878 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:32:52.710083 25893 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:52.710242 25878 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:52.710299 25878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2c44f78c107e49acaaa47f6c60392144"
format_stamp: "Formatted at 2026-03-19 16:32:52 on dist-test-slave-svn8"
server_key: "0eb5dcbb50e4cbc440a4732c04c83131"
server_key_iv: "1caafddc49987170e2f354b7485a9f47"
server_key_version: "encryptionkey@0"
I20260319 16:32:52.710350 25878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:52.724323 25878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:52.726112 25878 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:52.726210 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:52.729830 25878 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:38087
I20260319 16:32:52.729878 25945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:38087 every 8 connection(s)
I20260319 16:32:52.730142 25878 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:32:52.730672 25946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:52.732936 25946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144: Bootstrap starting.
I20260319 16:32:52.733568 25946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:52.733841 25946 log.cc:826] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:52.734488 25946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144: No bootstrap required, opened a new log
I20260319 16:32:52.735643 25946 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c44f78c107e49acaaa47f6c60392144" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38087 } }
I20260319 16:32:52.735771 25946 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:52.735798 25946 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c44f78c107e49acaaa47f6c60392144, State: Initialized, Role: FOLLOWER
I20260319 16:32:52.735908 25946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c44f78c107e49acaaa47f6c60392144" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38087 } }
I20260319 16:32:52.735975 25946 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:32:52.736003 25946 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:32:52.736044 25946 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:52.736544 25946 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c44f78c107e49acaaa47f6c60392144" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38087 } }
I20260319 16:32:52.736649 25946 leader_election.cc:304] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c44f78c107e49acaaa47f6c60392144; no voters:
I20260319 16:32:52.736788 25946 leader_election.cc:290] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:32:52.736837 25949 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:32:52.736977 25949 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [term 1 LEADER]: Becoming Leader. State: Replica: 2c44f78c107e49acaaa47f6c60392144, State: Running, Role: LEADER
I20260319 16:32:52.736994 25946 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:32:52.737059 25949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c44f78c107e49acaaa47f6c60392144" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38087 } }
I20260319 16:32:52.737422 25951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c44f78c107e49acaaa47f6c60392144. Latest consensus state: current_term: 1 leader_uuid: "2c44f78c107e49acaaa47f6c60392144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c44f78c107e49acaaa47f6c60392144" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38087 } } }
I20260319 16:32:52.737430 25950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c44f78c107e49acaaa47f6c60392144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c44f78c107e49acaaa47f6c60392144" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38087 } } }
I20260319 16:32:52.737510 25951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [sys.catalog]: This master's current role is: LEADER
I20260319 16:32:52.737604 25950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144 [sys.catalog]: This master's current role is: LEADER
I20260319 16:32:52.737946 25958 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:32:52.738370 25958 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:32:52.738848 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 25878
I20260319 16:32:52.738925 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:32:52.739086 25327 external_mini_cluster.cc:1468] Setting key 249ff6917acee1ee6a8e59062ee21b1b
I20260319 16:32:52.739821 25958 catalog_manager.cc:1357] Generated new cluster ID: 3147efd3e945458589f4df87eb4daa4b
I20260319 16:32:52.739871 25958 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:32:52 dist-test-slave-svn8 krb5kdc[25862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:52.751362 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:52.739474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.24.187.254:38087)
0319 16:32:52.739619 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:52.739630 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:52.739742 (+ 112us) server_negotiation.cc:408] Connection header received
0319 16:32:52.741848 (+ 2106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:52.741855 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:52.741901 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:52.742020 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:52.742939 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:52.744088 (+ 1149us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:52.744861 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:52.745082 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:52.747448 (+ 2366us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:52.747455 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:52.747456 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:52.747479 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:52.750015 (+ 2536us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:52.750355 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:52.750358 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:52.750358 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:52.750431 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:52.750569 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:52.750570 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:52.750570 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:52.750818 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:52.750960 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:52.751053 (+ 93us) server_negotiation.cc:300] Negotiation successful
0319 16:32:52.751180 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260319 16:32:52.758927 25958 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:32:52.759418 25958 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:32:52.769747 25958 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2c44f78c107e49acaaa47f6c60392144: Generated new TSK 0
I20260319 16:32:52.769927 25958 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:52.832969 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38087
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:52.910928 25974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:52.911082 25974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:52.911103 25974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:52.912487 25974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:52.912524 25974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:52.912549 25974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:32:52.914171 25974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:38087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:52.914331 25974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:52.914460 25974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:52.916759 25980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:52.916903 25974 server_base.cc:1060] running on GCE node
W20260319 16:32:52.916775 25982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:52.916759 25979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:52.917181 25974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:52.917369 25974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:52.918502 25974 hybrid_clock.cc:648] HybridClock initialized: now 1773937972918488 us; error 27 us; skew 500 ppm
Mar 19 16:32:52 dist-test-slave-svn8 krb5kdc[25862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937972, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:52.920730 25974 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:32:52.921195 25974 webserver.cc:492] Webserver started at http://127.24.187.193:42929/ using document root <none> and password file <none>
I20260319 16:32:52.921355 25974 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:52.921389 25974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:52.921500 25974 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:52.922464 25974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "ee05262afc3140e5a8f4b1f3b663dc08"
format_stamp: "Formatted at 2026-03-19 16:32:52 on dist-test-slave-svn8"
server_key: "472551a6a24af8bcfbcde12312c0856b"
server_key_iv: "841e2881e40bc9df69b7efb4344c4141"
server_key_version: "encryptionkey@0"
I20260319 16:32:52.922755 25974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "ee05262afc3140e5a8f4b1f3b663dc08"
format_stamp: "Formatted at 2026-03-19 16:32:52 on dist-test-slave-svn8"
server_key: "472551a6a24af8bcfbcde12312c0856b"
server_key_iv: "841e2881e40bc9df69b7efb4344c4141"
server_key_version: "encryptionkey@0"
I20260319 16:32:52.923863 25974 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:32:52.924558 25989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:52.924790 25974 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:32:52.924849 25974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "ee05262afc3140e5a8f4b1f3b663dc08"
format_stamp: "Formatted at 2026-03-19 16:32:52 on dist-test-slave-svn8"
server_key: "472551a6a24af8bcfbcde12312c0856b"
server_key_iv: "841e2881e40bc9df69b7efb4344c4141"
server_key_version: "encryptionkey@0"
I20260319 16:32:52.924909 25974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:52.950492 25974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:52.952347 25974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:52.952486 25974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:52.952688 25974 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:32:52.953001 25974 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:52.953032 25974 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:52.953060 25974 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:52.953078 25974 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:52.958590 25974 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:46545
I20260319 16:32:52.958650 26102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:46545 every 8 connection(s)
I20260319 16:32:52.958947 25974 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:32:52 dist-test-slave-svn8 krb5kdc[25862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937972, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:52.966830 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:52.959648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:54489 (local address 127.24.187.254:38087)
0319 16:32:52.959721 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:52.959723 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:52.959862 (+ 139us) server_negotiation.cc:408] Connection header received
0319 16:32:52.960371 (+ 509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:52.960373 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:52.960398 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:52.960450 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:52.961169 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:52.961698 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:52.962261 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:52.962436 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:52.964565 (+ 2129us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:52.964570 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:52.964571 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:52.964577 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:52.965678 (+ 1101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:52.966046 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:52.966047 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:52.966048 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:52.966075 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:52.966229 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:52.966230 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:52.966231 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:52.966336 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:52.966394 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:52.966624 (+ 230us) server_negotiation.cc:300] Negotiation successful
0319 16:32:52.966735 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260319 16:32:52.966904 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:52.959593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38087 (local address 127.24.187.193:54489)
0319 16:32:52.959753 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:52.959786 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:52.960280 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:52.960513 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:52.960516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:52.960798 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:52.961072 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:52.961076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:52.961798 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:52.961799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:52.962186 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:52.962187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:52.962299 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:52.963147 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:52.963161 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:52.964472 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:52.965769 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:52.965775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:52.965777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:52.965984 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:52.966124 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:52.966125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:52.966125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:52.966181 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:52.966389 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:52.966394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:52.966508 (+ 114us) client_negotiation.cc:769] Sending connection context
0319 16:32:52.966604 (+ 96us) client_negotiation.cc:240] Negotiation successful
0319 16:32:52.966730 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260319 16:32:52.967365 26103 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38087
I20260319 16:32:52.967468 26103 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:52.967670 26103 heartbeater.cc:507] Master 127.24.187.254:38087 requested a full tablet report, sending...
I20260319 16:32:52.968011 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 25974
I20260319 16:32:52.968087 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:32:52.968176 25910 ts_manager.cc:194] Registered new tserver with Master: ee05262afc3140e5a8f4b1f3b663dc08 (127.24.187.193:46545)
I20260319 16:32:52.968231 25327 external_mini_cluster.cc:1468] Setting key 6d0f7b8c8860d296d1e7cb0938eaaf41
I20260319 16:32:52.968945 25910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:54489
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:53.015884 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38087
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:53.089974 26110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:53.090137 26110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:53.090159 26110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:53.091585 26110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:53.091624 26110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:53.091647 26110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:32:53.093179 26110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:38087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:53.093359 26110 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:53.093518 26110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:53.095952 26116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:53.096041 26115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:53.096052 26118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:53.096159 26110 server_base.cc:1060] running on GCE node
I20260319 16:32:53.096320 26110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:53.096516 26110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:53.097663 26110 hybrid_clock.cc:648] HybridClock initialized: now 1773937973097624 us; error 52 us; skew 500 ppm
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:53.099987 26110 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:32:53.100518 26110 webserver.cc:492] Webserver started at http://127.24.187.194:45093/ using document root <none> and password file <none>
I20260319 16:32:53.100678 26110 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:53.100714 26110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:53.100836 26110 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:53.101766 26110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "9631c48f59834cd298d1b3625705e763"
format_stamp: "Formatted at 2026-03-19 16:32:53 on dist-test-slave-svn8"
server_key: "e1c9d8fe1c264da97b743101f266f43c"
server_key_iv: "12ed3561135c9d27f1881702847c2f55"
server_key_version: "encryptionkey@0"
I20260319 16:32:53.102068 26110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "9631c48f59834cd298d1b3625705e763"
format_stamp: "Formatted at 2026-03-19 16:32:53 on dist-test-slave-svn8"
server_key: "e1c9d8fe1c264da97b743101f266f43c"
server_key_iv: "12ed3561135c9d27f1881702847c2f55"
server_key_version: "encryptionkey@0"
I20260319 16:32:53.103163 26110 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:32:53.103884 26125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:53.104046 26110 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:32:53.104110 26110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "9631c48f59834cd298d1b3625705e763"
format_stamp: "Formatted at 2026-03-19 16:32:53 on dist-test-slave-svn8"
server_key: "e1c9d8fe1c264da97b743101f266f43c"
server_key_iv: "12ed3561135c9d27f1881702847c2f55"
server_key_version: "encryptionkey@0"
I20260319 16:32:53.104173 26110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:53.110589 26110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:53.112205 26110 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:53.112317 26110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:53.112473 26110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:32:53.112731 26110 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:53.112761 26110 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:53.112787 26110 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:53.112810 26110 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:53.118393 26110 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:36529
I20260319 16:32:53.118474 26238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:36529 every 8 connection(s)
I20260319 16:32:53.118710 26110 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:32:53.120123 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 26110
I20260319 16:32:53.120209 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:32:53.120353 25327 external_mini_cluster.cc:1468] Setting key cbe3f2d4360c6783515e1b2bd84cde16
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:53.126958 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.119269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38087 (local address 127.24.187.194:44157)
0319 16:32:53.119394 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:53.119423 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:53.119879 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:53.120088 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:53.120092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:53.120473 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:53.120784 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.120789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.121467 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.121468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:53.121991 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.121994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.122109 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.122996 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:53.123009 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:53.124237 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:53.125945 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:53.125951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:53.125953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:53.126139 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:53.126293 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:53.126293 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:53.126294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:53.126341 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:53.126557 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:53.126559 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:53.126655 (+ 96us) client_negotiation.cc:769] Sending connection context
0319 16:32:53.126737 (+ 82us) client_negotiation.cc:240] Negotiation successful
0319 16:32:53.126829 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260319 16:32:53.126955 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.119369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:44157 (local address 127.24.187.254:38087)
0319 16:32:53.119421 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.119422 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.119565 (+ 143us) server_negotiation.cc:408] Connection header received
0319 16:32:53.119947 (+ 382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.119949 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.119959 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.120020 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:53.120877 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.121380 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.122080 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.122213 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.124325 (+ 2112us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:53.124329 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:53.124329 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:53.124338 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:53.125854 (+ 1516us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.126210 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.126211 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.126211 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.126236 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.126393 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.126393 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.126394 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.126502 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:53.126564 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.126775 (+ 211us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.126864 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:32:53.127367 26239 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38087
I20260319 16:32:53.127475 26239 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:53.127692 26239 heartbeater.cc:507] Master 127.24.187.254:38087 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
I20260319 16:32:53.128104 25910 ts_manager.cc:194] Registered new tserver with Master: 9631c48f59834cd298d1b3625705e763 (127.24.187.194:36529)
I20260319 16:32:53.128468 25910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:44157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:53.166448 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38087
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:53.244887 26246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:53.245095 26246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:53.245131 26246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:53.247517 26246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:53.247578 26246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:53.247619 26246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:32:53.250240 26246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:38087
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:53.250507 26246 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:53.250700 26246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:53.253368 26251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:53.253358 26254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:53.253360 26252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:53.254033 26246 server_base.cc:1060] running on GCE node
I20260319 16:32:53.254185 26246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:53.254369 26246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:53.255512 26246 hybrid_clock.cc:648] HybridClock initialized: now 1773937973255489 us; error 34 us; skew 500 ppm
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:53.257915 26246 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:32:53.258361 26246 webserver.cc:492] Webserver started at http://127.24.187.195:44899/ using document root <none> and password file <none>
I20260319 16:32:53.258522 26246 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:53.258556 26246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:53.258664 26246 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:53.259531 26246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "15dc2c3877de49479df225d548ba51e5"
format_stamp: "Formatted at 2026-03-19 16:32:53 on dist-test-slave-svn8"
server_key: "b54e0be7bed2f9d9db8611c647fc0556"
server_key_iv: "2b8a5aa2e914f0a3a521fbf4f778eccf"
server_key_version: "encryptionkey@0"
I20260319 16:32:53.259825 26246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "15dc2c3877de49479df225d548ba51e5"
format_stamp: "Formatted at 2026-03-19 16:32:53 on dist-test-slave-svn8"
server_key: "b54e0be7bed2f9d9db8611c647fc0556"
server_key_iv: "2b8a5aa2e914f0a3a521fbf4f778eccf"
server_key_version: "encryptionkey@0"
I20260319 16:32:53.260963 26246 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:32:53.261598 26261 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:53.261773 26246 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:53.261831 26246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "15dc2c3877de49479df225d548ba51e5"
format_stamp: "Formatted at 2026-03-19 16:32:53 on dist-test-slave-svn8"
server_key: "b54e0be7bed2f9d9db8611c647fc0556"
server_key_iv: "2b8a5aa2e914f0a3a521fbf4f778eccf"
server_key_version: "encryptionkey@0"
I20260319 16:32:53.261891 26246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:53.281855 26246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:53.283646 26246 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:53.283747 26246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:53.283946 26246 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:32:53.284261 26246 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:53.284296 26246 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:53.284315 26246 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:53.284338 26246 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:53.290042 26246 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:32993
I20260319 16:32:53.290159 26374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:32993 every 8 connection(s)
I20260319 16:32:53.290382 26246 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:32:53.290923 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 26246
I20260319 16:32:53.291007 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:32:53.291170 25327 external_mini_cluster.cc:1468] Setting key 9f6421cd94f8d3f3f1ac3bec6dd62f7c
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:53.300355 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.291333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:36753 (local address 127.24.187.254:38087)
0319 16:32:53.291390 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.291392 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.291402 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:32:53.292086 (+ 684us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.292086 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.292098 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.292153 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:53.293142 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.293650 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.294465 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.294598 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.297472 (+ 2874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:53.297477 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:53.297478 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:53.297488 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:53.299137 (+ 1649us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.299551 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.299552 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.299552 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.299579 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.299752 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.299753 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.299753 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.299876 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:53.299932 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.300153 (+ 221us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.300255 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:32:53.300387 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.291022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38087 (local address 127.24.187.195:36753)
0319 16:32:53.291177 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:53.291264 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:53.291957 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:53.292223 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:53.292228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:53.292648 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:53.293052 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.293058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.293766 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.293767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:53.294361 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.294364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.294500 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.295504 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:53.295526 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:53.297346 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:53.299232 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:53.299239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:53.299241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:53.299478 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:53.299638 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:53.299639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:53.299639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:53.299704 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:53.299928 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:53.299931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:53.300043 (+ 112us) client_negotiation.cc:769] Sending connection context
0319 16:32:53.300130 (+ 87us) client_negotiation.cc:240] Negotiation successful
0319 16:32:53.300240 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260319 16:32:53.300909 26375 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38087
I20260319 16:32:53.301023 26375 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:53.301246 26375 heartbeater.cc:507] Master 127.24.187.254:38087 requested a full tablet report, sending...
I20260319 16:32:53.301674 25910 ts_manager.cc:194] Registered new tserver with Master: 15dc2c3877de49479df225d548ba51e5 (127.24.187.195:32993)
I20260319 16:32:53.302155 25910 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:36753
I20260319 16:32:53.302673 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:32:53.310250 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.304249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.24.187.254:38087)
0319 16:32:53.304308 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.304310 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.304426 (+ 116us) server_negotiation.cc:408] Connection header received
0319 16:32:53.304487 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.304488 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.304501 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.304569 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:53.305214 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.306016 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.306597 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.306761 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.307624 (+ 863us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:53.307628 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:53.307629 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:53.307642 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:53.309160 (+ 1518us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.309491 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.309492 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.309492 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.309514 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.309662 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.309663 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.309663 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.309829 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:53.309935 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.310060 (+ 125us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.310148 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:32:53.312119 25910 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:32:53.313372 25910 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:32:53.319403 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.315526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:36529 (local address 127.0.0.1:57806)
0319 16:32:53.315874 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:53.315903 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:53.315985 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:53.316445 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:53.316448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:53.316452 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:53.316725 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.316727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.318147 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.318148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:53.319102 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.319104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.319172 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.319213 (+ 41us) client_negotiation.cc:769] Sending connection context
0319 16:32:53.319258 (+ 45us) client_negotiation.cc:240] Negotiation successful
0319 16:32:53.319301 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":61,"threads_started":1}
I20260319 16:32:53.319972 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.315919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:32993 (local address 127.0.0.1:55764)
0319 16:32:53.316321 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:53.316330 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:53.316515 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:53.316745 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:53.316746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:53.316747 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:53.316966 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.316968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.318743 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.318744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:53.319786 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.319790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.319854 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.319857 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:32:53.319885 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:32:53.319919 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":66,"threads_started":1}
I20260319 16:32:53.320158 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.316157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.24.187.193:46545)
0319 16:32:53.316522 (+ 365us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.316528 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.316536 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:32:53.316570 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.316572 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.316588 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.316647 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:53.317050 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.318199 (+ 1149us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.319457 (+ 1258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.319963 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.320012 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.320044 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.320085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":102,"threads_started":1}
I20260319 16:32:53.320345 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.315636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.24.187.194:36529)
0319 16:32:53.315819 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.315826 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.315956 (+ 130us) server_negotiation.cc:408] Connection header received
0319 16:32:53.316083 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.316087 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.316107 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.316343 (+ 236us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:53.316821 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.318047 (+ 1226us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.319615 (+ 1568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.320159 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.320204 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.320232 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.320268 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260319 16:32:53.319404 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.315768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:46545 (local address 127.0.0.1:57254)
0319 16:32:53.315891 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:53.315907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:53.315979 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:53.316735 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:53.316736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:53.316738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:53.316966 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.316968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.318279 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.318280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:53.319177 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.319179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.319238 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.319242 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:32:53.319270 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:32:53.319308 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260319 16:32:53.320889 26173 tablet_service.cc:1511] Processing CreateTablet for tablet b5f7661005204bf0b435d2e1f4070c35 (DEFAULT_TABLE table=test-table [id=e3c8e6af84e14ecba4f39bd6165b2ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:53.321048 26037 tablet_service.cc:1511] Processing CreateTablet for tablet b5f7661005204bf0b435d2e1f4070c35 (DEFAULT_TABLE table=test-table [id=e3c8e6af84e14ecba4f39bd6165b2ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:53.321169 26173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f7661005204bf0b435d2e1f4070c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:53.321228 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.316163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.24.187.195:32993)
0319 16:32:53.316431 (+ 268us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.316439 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.316450 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:32:53.316597 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.316598 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.316614 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.316677 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:53.317047 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.318661 (+ 1614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.320382 (+ 1721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.321049 (+ 667us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.321088 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.321117 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.321157 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":86,"threads_started":1}
I20260319 16:32:53.321237 26037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f7661005204bf0b435d2e1f4070c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:53.321769 26309 tablet_service.cc:1511] Processing CreateTablet for tablet b5f7661005204bf0b435d2e1f4070c35 (DEFAULT_TABLE table=test-table [id=e3c8e6af84e14ecba4f39bd6165b2ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:53.321982 26309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f7661005204bf0b435d2e1f4070c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:53.323258 26392 tablet_bootstrap.cc:492] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08: Bootstrap starting.
I20260319 16:32:53.323879 26392 tablet_bootstrap.cc:654] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:53.324157 26392 log.cc:826] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:53.324498 26395 tablet_bootstrap.cc:492] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5: Bootstrap starting.
I20260319 16:32:53.324963 26395 tablet_bootstrap.cc:654] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:53.325205 26395 log.cc:826] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:53.325851 26393 tablet_bootstrap.cc:492] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763: Bootstrap starting.
I20260319 16:32:53.325974 26395 tablet_bootstrap.cc:492] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5: No bootstrap required, opened a new log
I20260319 16:32:53.326045 26395 ts_tablet_manager.cc:1403] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:32:53.326253 26392 tablet_bootstrap.cc:492] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08: No bootstrap required, opened a new log
I20260319 16:32:53.326318 26392 ts_tablet_manager.cc:1403] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260319 16:32:53.326697 26393 tablet_bootstrap.cc:654] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:53.327039 26393 log.cc:826] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:53.327548 26395 raft_consensus.cc:359] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.327697 26395 raft_consensus.cc:385] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:53.327725 26395 raft_consensus.cc:740] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dc2c3877de49479df225d548ba51e5, State: Initialized, Role: FOLLOWER
I20260319 16:32:53.327847 26395 consensus_queue.cc:260] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.327834 26392 raft_consensus.cc:359] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.327944 26392 raft_consensus.cc:385] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:53.327979 26392 raft_consensus.cc:740] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee05262afc3140e5a8f4b1f3b663dc08, State: Initialized, Role: FOLLOWER
I20260319 16:32:53.328064 26392 consensus_queue.cc:260] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.328102 26395 ts_tablet_manager.cc:1434] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:32:53.328202 26375 heartbeater.cc:499] Master 127.24.187.254:38087 was elected leader, sending a full tablet report...
I20260319 16:32:53.328276 26392 ts_tablet_manager.cc:1434] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:32:53.328397 26103 heartbeater.cc:499] Master 127.24.187.254:38087 was elected leader, sending a full tablet report...
I20260319 16:32:53.330312 26393 tablet_bootstrap.cc:492] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763: No bootstrap required, opened a new log
I20260319 16:32:53.330375 26393 ts_tablet_manager.cc:1403] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260319 16:32:53.331482 26393 raft_consensus.cc:359] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.331596 26393 raft_consensus.cc:385] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:53.331621 26393 raft_consensus.cc:740] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9631c48f59834cd298d1b3625705e763, State: Initialized, Role: FOLLOWER
I20260319 16:32:53.331796 26393 consensus_queue.cc:260] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.332080 26393 ts_tablet_manager.cc:1434] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:32:53.332147 26239 heartbeater.cc:499] Master 127.24.187.254:38087 was elected leader, sending a full tablet report...
W20260319 16:32:53.369360 26240 tablet.cc:2379] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:32:53.425026 26400 raft_consensus.cc:493] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:32:53.425208 26400 raft_consensus.cc:515] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.426043 26400 leader_election.cc:290] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee05262afc3140e5a8f4b1f3b663dc08 (127.24.187.193:46545), 15dc2c3877de49479df225d548ba51e5 (127.24.187.195:32993)
I20260319 16:32:53.429896 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.425801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:46545 (local address 127.24.187.194:41199)
0319 16:32:53.425849 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:53.425861 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:53.425922 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:53.426411 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:53.426412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:53.426413 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:53.426646 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.426648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.428763 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.428764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:53.429612 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.429614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.429734 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.429737 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:32:53.429773 (+ 36us) client_negotiation.cc:240] Negotiation successful
0319 16:32:53.429809 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260319 16:32:53.429905 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.425927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:32993 (local address 127.24.187.194:34813)
0319 16:32:53.426070 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:53.426079 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:53.426130 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:53.426368 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:53.426369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:53.426372 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:53.426672 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.426681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.428720 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.428722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:53.429633 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:53.429635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.429701 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.429741 (+ 40us) client_negotiation.cc:769] Sending connection context
0319 16:32:53.429773 (+ 32us) client_negotiation.cc:240] Negotiation successful
0319 16:32:53.429809 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260319 16:32:53.430485 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.426069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:41199 (local address 127.24.187.193:46545)
0319 16:32:53.426164 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.426166 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.426173 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:32:53.426201 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.426202 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.426210 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.426250 (+ 40us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:53.426744 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.427471 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.429745 (+ 2274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.430341 (+ 596us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.430353 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.430385 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.430421 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260319 16:32:53.430497 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.426055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:34813 (local address 127.24.187.195:32993)
0319 16:32:53.426124 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.426126 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.426134 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:32:53.426309 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.426310 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.426318 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.426363 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:53.427570 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.428591 (+ 1021us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.429748 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.430362 (+ 614us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.430371 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.430391 (+ 20us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.430442 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260319 16:32:53.431210 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5f7661005204bf0b435d2e1f4070c35" candidate_uuid: "9631c48f59834cd298d1b3625705e763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" is_pre_election: true
I20260319 16:32:53.431219 26329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5f7661005204bf0b435d2e1f4070c35" candidate_uuid: "9631c48f59834cd298d1b3625705e763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dc2c3877de49479df225d548ba51e5" is_pre_election: true
I20260319 16:32:53.431368 26329 raft_consensus.cc:2468] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9631c48f59834cd298d1b3625705e763 in term 0.
I20260319 16:32:53.431370 26057 raft_consensus.cc:2468] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9631c48f59834cd298d1b3625705e763 in term 0.
I20260319 16:32:53.431613 26128 leader_election.cc:304] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15dc2c3877de49479df225d548ba51e5, 9631c48f59834cd298d1b3625705e763; no voters:
I20260319 16:32:53.431767 26400 raft_consensus.cc:2804] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:32:53.431826 26400 raft_consensus.cc:493] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:32:53.431854 26400 raft_consensus.cc:3060] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:53.432812 26400 raft_consensus.cc:515] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.432955 26400 leader_election.cc:290] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [CANDIDATE]: Term 1 election: Requested vote from peers ee05262afc3140e5a8f4b1f3b663dc08 (127.24.187.193:46545), 15dc2c3877de49479df225d548ba51e5 (127.24.187.195:32993)
I20260319 16:32:53.433128 26329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5f7661005204bf0b435d2e1f4070c35" candidate_uuid: "9631c48f59834cd298d1b3625705e763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dc2c3877de49479df225d548ba51e5"
I20260319 16:32:53.433128 26057 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5f7661005204bf0b435d2e1f4070c35" candidate_uuid: "9631c48f59834cd298d1b3625705e763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee05262afc3140e5a8f4b1f3b663dc08"
I20260319 16:32:53.433214 26057 raft_consensus.cc:3060] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:53.433214 26329 raft_consensus.cc:3060] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:53.434111 26329 raft_consensus.cc:2468] T b5f7661005204bf0b435d2e1f4070c35 P 15dc2c3877de49479df225d548ba51e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9631c48f59834cd298d1b3625705e763 in term 1.
I20260319 16:32:53.434172 26057 raft_consensus.cc:2468] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9631c48f59834cd298d1b3625705e763 in term 1.
I20260319 16:32:53.434340 26128 leader_election.cc:304] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9631c48f59834cd298d1b3625705e763, ee05262afc3140e5a8f4b1f3b663dc08; no voters:
I20260319 16:32:53.434427 26400 raft_consensus.cc:2804] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:32:53.434563 26400 raft_consensus.cc:697] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [term 1 LEADER]: Becoming Leader. State: Replica: 9631c48f59834cd298d1b3625705e763, State: Running, Role: LEADER
I20260319 16:32:53.434656 26400 consensus_queue.cc:237] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } }
I20260319 16:32:53.435446 25909 catalog_manager.cc:5671] T b5f7661005204bf0b435d2e1f4070c35 P 9631c48f59834cd298d1b3625705e763 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9631c48f59834cd298d1b3625705e763 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "9631c48f59834cd298d1b3625705e763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9631c48f59834cd298d1b3625705e763" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee05262afc3140e5a8f4b1f3b663dc08" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 46545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15dc2c3877de49479df225d548ba51e5" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 32993 } health_report { overall_health: UNKNOWN } } }
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:32:53.443975 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.437409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.24.187.193:46545)
0319 16:32:53.437456 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.437458 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.437576 (+ 118us) server_negotiation.cc:408] Connection header received
0319 16:32:53.437641 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.437642 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.437654 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.437731 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:53.438429 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.438920 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.439410 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.439558 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.441566 (+ 2008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:53.441573 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:53.441574 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:53.441579 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:53.443074 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.443369 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.443372 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.443373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.443412 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.443544 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.443544 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.443545 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.443630 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:53.443774 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.443810 (+ 36us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.443891 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260319 16:32:53.459586 26104 tablet.cc:2379] T b5f7661005204bf0b435d2e1f4070c35 P ee05262afc3140e5a8f4b1f3b663dc08: Can't schedule compaction. Clean time has not been advanced past its initial value.
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:32:53.465384 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.458696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.24.187.193:46545)
0319 16:32:53.458760 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.458762 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.458787 (+ 25us) server_negotiation.cc:408] Connection header received
0319 16:32:53.458852 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.458863 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.458874 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.458917 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:53.459623 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.460113 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.460789 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.460917 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.463175 (+ 2258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:53.463179 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:53.463179 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:53.463187 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:53.464561 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.464828 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.464829 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.464830 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.464849 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.464995 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.464995 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.464995 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.465064 (+ 69us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:53.465124 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.465253 (+ 129us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.465316 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[25862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:32:53.487413 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:53.480945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.24.187.193:46545)
0319 16:32:53.481055 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:53.481057 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:53.481167 (+ 110us) server_negotiation.cc:408] Connection header received
0319 16:32:53.481199 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:53.481200 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:53.481210 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:53.481265 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:53.481981 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.482483 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:53.483076 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:53.483229 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:53.485190 (+ 1961us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:53.485195 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:53.485196 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:53.485204 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:53.486578 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.486861 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.486861 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.486861 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.486878 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:53.486982 (+ 104us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:53.486982 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:53.486982 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:53.487050 (+ 68us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:53.487119 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:53.487255 (+ 136us) server_negotiation.cc:300] Negotiation successful
0319 16:32:53.487312 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260319 16:32:53.487807 26017 tablet_service.cc:3018] Rejecting scan request for tablet b5f7661005204bf0b435d2e1f4070c35: Uninitialized: clean time has not yet been initialized
I20260319 16:32:53.488480 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 25974
I20260319 16:32:53.493516 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 26110
I20260319 16:32:53.497767 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 26246
I20260319 16:32:53.501967 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 25878
2026-03-19T16:32:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2860 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[26416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[26416](info): set up 2 sockets
Mar 19 16:32:53 dist-test-slave-svn8 krb5kdc[26416](info): commencing operation
krb5kdc: starting...
W20260319 16:32:55.358350 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.836s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:32:55 dist-test-slave-svn8 krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:32:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:55.489786 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37091
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:37091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:55.563344 26432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:55.563522 26432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:55.563544 26432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:55.565006 26432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:32:55.565049 26432 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260319 16:32:55.565061 26432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:55.565071 26432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:55.565083 26432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:32:55.565092 26432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:32:55.566735 26432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:37091
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:55.566919 26432 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:55.567080 26432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:55.569209 26438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:55.569283 26440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:55.569283 26437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:55.569461 26432 server_base.cc:1060] running on GCE node
I20260319 16:32:55.569667 26432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:55.569919 26432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:55.571069 26432 hybrid_clock.cc:648] HybridClock initialized: now 1773937975571041 us; error 36 us; skew 500 ppm
Mar 19 16:32:55 dist-test-slave-svn8 krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:55.573379 26432 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:32:55.573911 26432 webserver.cc:492] Webserver started at http://127.24.187.254:34025/ using document root <none> and password file <none>
I20260319 16:32:55.574070 26432 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:55.574102 26432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:55.574201 26432 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:55.575086 26432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "48e2e5682f83453ca7e90b48d3255f28"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "532254af413a6845f681b00bed122bca"
server_key_iv: "dd39f5178ce48565027fcf13569530c2"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.575377 26432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "48e2e5682f83453ca7e90b48d3255f28"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "532254af413a6845f681b00bed122bca"
server_key_iv: "dd39f5178ce48565027fcf13569530c2"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.576490 26432 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:32:55.577158 26447 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:55.577404 26432 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:32:55.577463 26432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "48e2e5682f83453ca7e90b48d3255f28"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "532254af413a6845f681b00bed122bca"
server_key_iv: "dd39f5178ce48565027fcf13569530c2"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.577523 26432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:55.587059 26432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:55.594364 26432 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:55.594487 26432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:55.598845 26432 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:37091
I20260319 16:32:55.598881 26509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:37091 every 8 connection(s)
I20260319 16:32:55.599170 26432 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:32:55.599718 26510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:55.602126 26510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28: Bootstrap starting.
I20260319 16:32:55.602681 26510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:55.602902 26510 log.cc:826] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:55.603533 26510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28: No bootstrap required, opened a new log
I20260319 16:32:55.604046 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 26432
I20260319 16:32:55.604127 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:32:55.604270 25327 external_mini_cluster.cc:1468] Setting key 79087e856b10426fdcab9a21c73801e0
I20260319 16:32:55.604998 26510 raft_consensus.cc:359] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e2e5682f83453ca7e90b48d3255f28" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37091 } }
I20260319 16:32:55.605114 26510 raft_consensus.cc:385] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:55.605134 26510 raft_consensus.cc:740] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e2e5682f83453ca7e90b48d3255f28, State: Initialized, Role: FOLLOWER
I20260319 16:32:55.605221 26510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e2e5682f83453ca7e90b48d3255f28" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37091 } }
I20260319 16:32:55.605278 26510 raft_consensus.cc:399] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:32:55.605300 26510 raft_consensus.cc:493] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:32:55.605342 26510 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:55.605885 26510 raft_consensus.cc:515] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e2e5682f83453ca7e90b48d3255f28" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37091 } }
I20260319 16:32:55.605978 26510 leader_election.cc:304] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e2e5682f83453ca7e90b48d3255f28; no voters:
I20260319 16:32:55.606109 26510 leader_election.cc:290] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:32:55.606153 26515 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:32:55.606295 26515 raft_consensus.cc:697] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [term 1 LEADER]: Becoming Leader. State: Replica: 48e2e5682f83453ca7e90b48d3255f28, State: Running, Role: LEADER
I20260319 16:32:55.606314 26510 sys_catalog.cc:565] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:32:55.606375 26515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e2e5682f83453ca7e90b48d3255f28" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37091 } }
I20260319 16:32:55.606694 26515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e2e5682f83453ca7e90b48d3255f28. Latest consensus state: current_term: 1 leader_uuid: "48e2e5682f83453ca7e90b48d3255f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e2e5682f83453ca7e90b48d3255f28" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37091 } } }
I20260319 16:32:55.606762 26515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [sys.catalog]: This master's current role is: LEADER
I20260319 16:32:55.606706 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48e2e5682f83453ca7e90b48d3255f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e2e5682f83453ca7e90b48d3255f28" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37091 } } }
I20260319 16:32:55.606794 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28 [sys.catalog]: This master's current role is: LEADER
I20260319 16:32:55.607187 26519 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:32:55.607645 26519 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:32:55.609337 26519 catalog_manager.cc:1357] Generated new cluster ID: cda3817204274dbcaf5360ba7de2ca5f
I20260319 16:32:55.609385 26519 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:32:55 dist-test-slave-svn8 krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:55.613647 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.604756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.24.187.254:37091)
0319 16:32:55.604951 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:55.604960 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:55.604977 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:32:55.605062 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:55.605067 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:55.605101 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:55.605208 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:55.606086 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.607178 (+ 1092us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.608191 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.609074 (+ 883us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.610546 (+ 1472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:55.610551 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:55.610551 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:55.610569 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:55.612347 (+ 1778us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.612696 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.612698 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.612699 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.612759 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.612900 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.612901 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.612901 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.613096 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:55.613214 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:55.613359 (+ 145us) server_negotiation.cc:300] Negotiation successful
0319 16:32:55.613471 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260319 16:32:55.618363 26519 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:32:55.618877 26519 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:32:55.626567 26519 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28: Generated new TSK 0
I20260319 16:32:55.626754 26519 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:55.674985 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37091
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:55.750270 26538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:55.750439 26538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:55.750464 26538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:55.751924 26538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:55.751969 26538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:55.751989 26538 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:32:55.752005 26538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:32:55.753791 26538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:37091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:55.754081 26538 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:55.754268 26538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:55.756682 26543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:55.756735 26546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:55.756731 26544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:55.756911 26538 server_base.cc:1060] running on GCE node
I20260319 16:32:55.757047 26538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:55.757220 26538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:55.758347 26538 hybrid_clock.cc:648] HybridClock initialized: now 1773937975758327 us; error 29 us; skew 500 ppm
Mar 19 16:32:55 dist-test-slave-svn8 krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:55.760833 26538 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:32:55.761318 26538 webserver.cc:492] Webserver started at http://127.24.187.193:38767/ using document root <none> and password file <none>
I20260319 16:32:55.761479 26538 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:55.761515 26538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:55.761615 26538 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:55.762554 26538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "59c9efa1c672372ec625f02a3ec8235d"
server_key_iv: "0f53b44816f451bb9d9c693b186f7244"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.762853 26538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "59c9efa1c672372ec625f02a3ec8235d"
server_key_iv: "0f53b44816f451bb9d9c693b186f7244"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.763932 26538 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:32:55.764683 26553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:55.764895 26538 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:32:55.764961 26538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "59c9efa1c672372ec625f02a3ec8235d"
server_key_iv: "0f53b44816f451bb9d9c693b186f7244"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.765021 26538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:55.786571 26538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:55.788481 26538 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:55.788612 26538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:55.789173 26538 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:55.789218 26538 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:55.789249 26538 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:55.789273 26538 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 19 16:32:55 dist-test-slave-svn8 krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:55.796631 26538 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:37033
I20260319 16:32:55.796844 26668 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:37033 every 8 connection(s)
I20260319 16:32:55.796990 26538 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:32:55.798498 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.789495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:44335 (local address 127.24.187.254:37091)
0319 16:32:55.789565 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:55.789567 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:55.789746 (+ 179us) server_negotiation.cc:408] Connection header received
0319 16:32:55.790514 (+ 768us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:55.790516 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:55.790534 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:55.790606 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:55.791568 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.792323 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.793119 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.793327 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.795681 (+ 2354us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:55.795687 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:55.795687 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:55.795697 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:55.797370 (+ 1673us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.797801 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.797802 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.797803 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.797833 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.797987 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.797987 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.797988 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.798082 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:55.798131 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:55.798334 (+ 203us) server_negotiation.cc:300] Negotiation successful
0319 16:32:55.798433 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:32:55.798522 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.789467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37091 (local address 127.24.187.193:44335)
0319 16:32:55.789609 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:55.789642 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:55.790307 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:55.790673 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:55.790677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:55.791041 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:55.791448 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.791454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.792436 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.792437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:55.793012 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.793016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.793116 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.793624 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:55.793639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:55.795566 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:55.797447 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.797451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.797453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.797702 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.797893 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.797894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.797894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.797945 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.798124 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:55.798126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:55.798225 (+ 99us) client_negotiation.cc:769] Sending connection context
0319 16:32:55.798298 (+ 73us) client_negotiation.cc:240] Negotiation successful
0319 16:32:55.798394 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260319 16:32:55.799118 26669 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37091
I20260319 16:32:55.799206 26669 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:55.799415 26669 heartbeater.cc:507] Master 127.24.187.254:37091 requested a full tablet report, sending...
I20260319 16:32:55.799425 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 26538
I20260319 16:32:55.799494 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:32:55.799679 25327 external_mini_cluster.cc:1468] Setting key 73e3c58bec581d04ec0fda0014e20977
I20260319 16:32:55.799921 26463 ts_manager.cc:194] Registered new tserver with Master: 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033)
I20260319 16:32:55.800724 26463 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:44335
I20260319 16:32:55.805397 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.800262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:41357 (local address 127.24.187.254:37091)
0319 16:32:55.800313 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:55.800315 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:55.800651 (+ 336us) server_negotiation.cc:408] Connection header received
0319 16:32:55.800751 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:55.800752 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:55.800763 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:55.800791 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:55.801265 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.801756 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.802378 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.802543 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.803286 (+ 743us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:55.803289 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:55.803289 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:55.803295 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:55.804482 (+ 1187us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.804829 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.804830 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.804831 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.804855 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.805003 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.805004 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.805004 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.805081 (+ 77us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:55.805123 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:55.805259 (+ 136us) server_negotiation.cc:300] Negotiation successful
0319 16:32:55.805329 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260319 16:32:55.805454 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.800460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37091 (local address 127.24.187.193:41357)
0319 16:32:55.800593 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:55.800604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:55.800676 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:55.800852 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:55.800853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:55.801011 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:55.801189 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.801191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.801820 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.801820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:55.802273 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.802277 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.802358 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.802900 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:55.802905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:55.803187 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:55.804559 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.804560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.804561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.804741 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.804920 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.804921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.804921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.804953 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.805128 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:55.805129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:55.805175 (+ 46us) client_negotiation.cc:769] Sending connection context
0319 16:32:55.805246 (+ 71us) client_negotiation.cc:240] Negotiation successful
0319 16:32:55.805348 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:55.845037 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37091
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:55.918536 26680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:55.918697 26680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:55.918726 26680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:55.920146 26680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:55.920190 26680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:55.920217 26680 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:32:55.920239 26680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:32:55.921865 26680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:37091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:55.922070 26680 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:55.922217 26680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:55.924466 26686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:55.924540 26688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:55.924543 26685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:55.924665 26680 server_base.cc:1060] running on GCE node
I20260319 16:32:55.924809 26680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:55.924997 26680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:55.926115 26680 hybrid_clock.cc:648] HybridClock initialized: now 1773937975926099 us; error 32 us; skew 500 ppm
Mar 19 16:32:55 dist-test-slave-svn8 krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:55.928368 26680 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:32:55.928862 26680 webserver.cc:492] Webserver started at http://127.24.187.194:43649/ using document root <none> and password file <none>
I20260319 16:32:55.929035 26680 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:55.929069 26680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:55.929159 26680 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:55.930094 26680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "5f00f007fbbe4b55a087b6eb4253f334"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "3beb037c57077912f64ad296d3447d67"
server_key_iv: "29c3ebb2b5e84ccd36e422f49697aac1"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.930377 26680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "5f00f007fbbe4b55a087b6eb4253f334"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "3beb037c57077912f64ad296d3447d67"
server_key_iv: "29c3ebb2b5e84ccd36e422f49697aac1"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.931439 26680 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:32:55.932047 26695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:55.932199 26680 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:32:55.932255 26680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "5f00f007fbbe4b55a087b6eb4253f334"
format_stamp: "Formatted at 2026-03-19 16:32:55 on dist-test-slave-svn8"
server_key: "3beb037c57077912f64ad296d3447d67"
server_key_iv: "29c3ebb2b5e84ccd36e422f49697aac1"
server_key_version: "encryptionkey@0"
I20260319 16:32:55.932313 26680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:55.942793 26680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:55.944388 26680 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:55.944502 26680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:55.944983 26680 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:55.945019 26680 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:55.945045 26680 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:55.945062 26680 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 19 16:32:55 dist-test-slave-svn8 krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:55.951799 26680 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:46485
I20260319 16:32:55.951836 26810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:46485 every 8 connection(s)
I20260319 16:32:55.952152 26680 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:32:55.952907 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.945305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:49441 (local address 127.24.187.254:37091)
0319 16:32:55.945370 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:55.945373 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:55.945500 (+ 127us) server_negotiation.cc:408] Connection header received
0319 16:32:55.946010 (+ 510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:55.946012 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:55.946027 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:55.946103 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:55.946857 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.947386 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.947898 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.948064 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.950033 (+ 1969us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:55.950037 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:55.950037 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:55.950045 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:55.951658 (+ 1613us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.952029 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.952031 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.952031 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.952060 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.952254 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.952254 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.952255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.952377 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:55.952453 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:55.952751 (+ 298us) server_negotiation.cc:300] Negotiation successful
0319 16:32:55.952841 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260319 16:32:55.952936 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.945260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37091 (local address 127.24.187.194:49441)
0319 16:32:55.945397 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:55.945426 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:55.945921 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:55.946155 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:55.946159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:55.946494 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:55.946763 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.946767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.947458 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.947459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:55.947827 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.947828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.947932 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.948369 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:55.948391 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:55.949952 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:55.951735 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.951738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.951739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.951955 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.952134 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.952135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.952135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.952188 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.952485 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:55.952489 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:55.952624 (+ 135us) client_negotiation.cc:769] Sending connection context
0319 16:32:55.952712 (+ 88us) client_negotiation.cc:240] Negotiation successful
0319 16:32:55.952812 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260319 16:32:55.953338 26811 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37091
I20260319 16:32:55.953416 26811 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:55.953585 26811 heartbeater.cc:507] Master 127.24.187.254:37091 requested a full tablet report, sending...
I20260319 16:32:55.953965 26464 ts_manager.cc:194] Registered new tserver with Master: 5f00f007fbbe4b55a087b6eb4253f334 (127.24.187.194:46485)
I20260319 16:32:55.954394 26464 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:49441
I20260319 16:32:55.958914 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.954271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37091 (local address 127.24.187.194:34297)
0319 16:32:55.954388 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:55.954399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:55.954470 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:55.954648 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:55.954649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:55.954803 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:55.955024 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.955027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.955599 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.955600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:55.955958 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:55.955959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.956014 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.956382 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:55.956386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:55.956611 (+ 225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:55.958052 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.958053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.958053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.958186 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.958356 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:55.958357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:55.958357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:55.958394 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:55.958627 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:55.958629 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:55.958672 (+ 43us) client_negotiation.cc:769] Sending connection context
0319 16:32:55.958747 (+ 75us) client_negotiation.cc:240] Negotiation successful
0319 16:32:55.958829 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260319 16:32:55.958931 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:55.954352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:34297 (local address 127.24.187.254:37091)
0319 16:32:55.954388 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:55.954390 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:55.954455 (+ 65us) server_negotiation.cc:408] Connection header received
0319 16:32:55.954564 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:55.954565 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:55.954576 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:55.954630 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:55.955092 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.955536 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:55.956014 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:55.956112 (+ 98us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:55.956669 (+ 557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:55.956672 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:55.956672 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:55.956676 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:55.957966 (+ 1290us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.958276 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.958277 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.958278 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.958301 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:55.958436 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:55.958437 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:55.958437 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:55.958544 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:55.958595 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:55.958801 (+ 206us) server_negotiation.cc:300] Negotiation successful
0319 16:32:55.958872 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:32:55.959408 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 26680
I20260319 16:32:55.959510 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:32:55.959708 25327 external_mini_cluster.cc:1468] Setting key 11c129567d2d5338dc60f8bcf96e574d
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:32:56.004667 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37091
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:32:56.078418 26822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:32:56.078612 26822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:32:56.078636 26822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:32:56.080088 26822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:32:56.080135 26822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:32:56.080155 26822 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:32:56.080171 26822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:32:56.081876 26822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:37091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:32:56.082075 26822 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:32:56.082237 26822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:32:56.084430 26830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:56.084589 26827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:32:56.084728 26828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:32:56.084959 26822 server_base.cc:1060] running on GCE node
I20260319 16:32:56.085093 26822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:32:56.085265 26822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:32:56.086408 26822 hybrid_clock.cc:648] HybridClock initialized: now 1773937976086387 us; error 30 us; skew 500 ppm
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937976, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:32:56.088909 26822 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:32:56.089443 26822 webserver.cc:492] Webserver started at http://127.24.187.195:39329/ using document root <none> and password file <none>
I20260319 16:32:56.089601 26822 fs_manager.cc:362] Metadata directory not provided
I20260319 16:32:56.089635 26822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:32:56.089745 26822 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:32:56.090667 26822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "7136f2d91c7d4f4b9ec872170ae241d6"
format_stamp: "Formatted at 2026-03-19 16:32:56 on dist-test-slave-svn8"
server_key: "a4b21663fdb4d068780d843d937d03cb"
server_key_iv: "448ead6409bd7074b9cf959aefad2a76"
server_key_version: "encryptionkey@0"
I20260319 16:32:56.090948 26822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "7136f2d91c7d4f4b9ec872170ae241d6"
format_stamp: "Formatted at 2026-03-19 16:32:56 on dist-test-slave-svn8"
server_key: "a4b21663fdb4d068780d843d937d03cb"
server_key_iv: "448ead6409bd7074b9cf959aefad2a76"
server_key_version: "encryptionkey@0"
I20260319 16:32:56.092053 26822 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:32:56.092737 26837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:56.092916 26822 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:32:56.092974 26822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "7136f2d91c7d4f4b9ec872170ae241d6"
format_stamp: "Formatted at 2026-03-19 16:32:56 on dist-test-slave-svn8"
server_key: "a4b21663fdb4d068780d843d937d03cb"
server_key_iv: "448ead6409bd7074b9cf959aefad2a76"
server_key_version: "encryptionkey@0"
I20260319 16:32:56.093024 26822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:32:56.113049 26822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:32:56.114898 26822 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:32:56.114995 26822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:32:56.115432 26822 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:32:56.115465 26822 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:56.115486 26822 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:32:56.115499 26822 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:56.121639 26822 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:39987
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937976, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:56.121865 26952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:39987 every 8 connection(s)
I20260319 16:32:56.121999 26822 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:32:56.124579 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.115844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:52357 (local address 127.24.187.254:37091)
0319 16:32:56.115907 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.115909 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.116101 (+ 192us) server_negotiation.cc:408] Connection header received
0319 16:32:56.116610 (+ 509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.116612 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.116626 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.116702 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:56.117642 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.118397 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.119213 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.119401 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.122298 (+ 2897us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:56.122302 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:56.122303 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:56.122310 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:56.123531 (+ 1221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.123858 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.123859 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.123859 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.123887 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.124042 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.124043 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.124044 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.124134 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:56.124188 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.124409 (+ 221us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.124491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:32:56.124677 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.115803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37091 (local address 127.24.187.195:52357)
0319 16:32:56.115986 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.116020 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.116518 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.116764 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.116768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.117138 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:56.117533 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.117538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.118540 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.118541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.119130 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.119133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.119265 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.119809 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:56.119833 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:56.122189 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:56.123609 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:56.123614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:56.123616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:56.123795 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:56.123935 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:56.123936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:56.123937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:56.123991 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:56.124184 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:56.124189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:56.124294 (+ 105us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.124390 (+ 96us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.124512 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260319 16:32:56.125087 26953 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37091
I20260319 16:32:56.125169 26953 heartbeater.cc:461] Registering TS with master...
I20260319 16:32:56.125375 26953 heartbeater.cc:507] Master 127.24.187.254:37091 requested a full tablet report, sending...
I20260319 16:32:56.125808 26463 ts_manager.cc:194] Registered new tserver with Master: 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195:39987)
I20260319 16:32:56.126281 26463 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:52357
I20260319 16:32:56.129318 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 26822
I20260319 16:32:56.129400 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:32:56.129590 25327 external_mini_cluster.cc:1468] Setting key 8e983c49d79efa425227ae17b95729e1
I20260319 16:32:56.130376 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:32:56.130873 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.126138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37091 (local address 127.24.187.195:51431)
0319 16:32:56.126269 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.126281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.126335 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.126502 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.126503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.126652 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:56.126846 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.126849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.127507 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.127508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.127870 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.127872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.127928 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.128315 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:56.128319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:56.128586 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:56.130122 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:56.130123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:56.130124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:56.130254 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:56.130389 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:56.130390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:56.130390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:56.130429 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:56.130633 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:56.130634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:56.130674 (+ 40us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.130719 (+ 45us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.130797 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260319 16:32:56.130906 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.126198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:51431 (local address 127.24.187.254:37091)
0319 16:32:56.126402 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.126404 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.126412 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:32:56.126429 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.126430 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.126439 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.126496 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:56.126915 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.127436 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.127934 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.128087 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.128677 (+ 590us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:56.128679 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:56.128680 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:56.128686 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:56.130051 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.130310 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.130312 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.130312 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.130334 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.130476 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.130477 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.130477 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.130573 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:56.130629 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.130752 (+ 123us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.130821 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[26416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:56.151209 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.144978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.24.187.254:37091)
0319 16:32:56.145034 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.145036 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.145045 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:32:56.145080 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.145081 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.145089 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.145139 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:56.145910 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.146384 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.146974 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.147097 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.149135 (+ 2038us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:56.149140 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:56.149140 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:56.149147 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:56.150328 (+ 1181us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.150633 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.150634 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.150634 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.150650 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.150752 (+ 102us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.150752 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.150752 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.150833 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:56.150905 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.151069 (+ 164us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.151139 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:32:56.152638 26463 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:32:56.153959 26463 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:32:56.159324 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.155963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46485 (local address 127.0.0.1:35788)
0319 16:32:56.156129 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.156161 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.156235 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.156659 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.156663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.156666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:56.156956 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.156959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.158317 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.158318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.159143 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.159145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.159205 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.159207 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.159233 (+ 26us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.159262 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260319 16:32:56.159564 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.156191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:39987 (local address 127.0.0.1:57502)
0319 16:32:56.156641 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.156653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.156701 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.157028 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.157029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.157031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:56.157228 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.157230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.158471 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.158472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.159181 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.159184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.159246 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.159250 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.159278 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.159317 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":67,"threads_started":1}
I20260319 16:32:56.159965 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.156483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.24.187.193:37033)
0319 16:32:56.156704 (+ 221us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.156711 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.156721 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:32:56.156759 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.156761 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.156776 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.156880 (+ 104us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:56.157246 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.158019 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.159322 (+ 1303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.159816 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.159857 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.159883 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.159911 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260319 16:32:56.159240 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.155975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:37033 (local address 127.0.0.1:60034)
0319 16:32:56.156300 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.156311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.156375 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.156985 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.156986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.156988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:56.157165 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.157168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.158115 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.158116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.158964 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.158967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.159032 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.159076 (+ 44us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.159118 (+ 42us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.159159 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":94,"threads_started":1}
I20260319 16:32:56.160053 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.156284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.24.187.195:39987)
0319 16:32:56.156800 (+ 516us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.156805 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.156813 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:32:56.156845 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.156852 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.156867 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.156925 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:56.157296 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.158394 (+ 1098us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.159373 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.159862 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.159910 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.159938 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.159977 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":107,"threads_started":1}
I20260319 16:32:56.160404 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.156325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.24.187.194:46485)
0319 16:32:56.156465 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.156472 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.156483 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:32:56.156520 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.156522 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.156538 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.156613 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:56.157053 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.158226 (+ 1173us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.159242 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.160067 (+ 825us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.160104 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.160303 (+ 199us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.160348 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260319 16:32:56.160601 26603 tablet_service.cc:1511] Processing CreateTablet for tablet 1d046ab17fc84c7f8601f7f52d211a23 (DEFAULT_TABLE table=test-table [id=67b163a4918f4729adf30687a3e13323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:56.160861 26603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d046ab17fc84c7f8601f7f52d211a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.160863 26745 tablet_service.cc:1511] Processing CreateTablet for tablet 1d046ab17fc84c7f8601f7f52d211a23 (DEFAULT_TABLE table=test-table [id=67b163a4918f4729adf30687a3e13323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:56.161113 26745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d046ab17fc84c7f8601f7f52d211a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.162029 26887 tablet_service.cc:1511] Processing CreateTablet for tablet 1d046ab17fc84c7f8601f7f52d211a23 (DEFAULT_TABLE table=test-table [id=67b163a4918f4729adf30687a3e13323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:56.162262 26887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d046ab17fc84c7f8601f7f52d211a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.163177 26976 tablet_bootstrap.cc:492] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Bootstrap starting.
I20260319 16:32:56.163542 26977 tablet_bootstrap.cc:492] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Bootstrap starting.
I20260319 16:32:56.163923 26976 tablet_bootstrap.cc:654] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.164084 26977 tablet_bootstrap.cc:654] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.164510 26978 tablet_bootstrap.cc:492] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Bootstrap starting.
I20260319 16:32:56.164546 26976 log.cc:826] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:56.164776 26977 log.cc:826] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:56.165040 26978 tablet_bootstrap.cc:654] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.165297 26978 log.cc:826] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Log is configured to *not* fsync() on all Append() calls
I20260319 16:32:56.165325 26976 tablet_bootstrap.cc:492] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: No bootstrap required, opened a new log
I20260319 16:32:56.165349 26977 tablet_bootstrap.cc:492] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: No bootstrap required, opened a new log
I20260319 16:32:56.165378 26976 ts_tablet_manager.cc:1403] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:32:56.165400 26977 ts_tablet_manager.cc:1403] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:32:56.166102 26978 tablet_bootstrap.cc:492] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: No bootstrap required, opened a new log
I20260319 16:32:56.166158 26978 ts_tablet_manager.cc:1403] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:32:56.166909 26977 raft_consensus.cc:359] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.166910 26976 raft_consensus.cc:359] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.167052 26976 raft_consensus.cc:385] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.167052 26977 raft_consensus.cc:385] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.167080 26976 raft_consensus.cc:740] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f00f007fbbe4b55a087b6eb4253f334, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.167080 26977 raft_consensus.cc:740] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9fcfdd4e7e4c2e8a09e6326e9ed12a, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.167167 26976 consensus_queue.cc:260] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.167207 26977 consensus_queue.cc:260] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.167440 26976 ts_tablet_manager.cc:1434] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:32:56.167443 26977 ts_tablet_manager.cc:1434] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:32:56.167392 26978 raft_consensus.cc:359] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.167516 26978 raft_consensus.cc:385] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.167549 26978 raft_consensus.cc:740] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7136f2d91c7d4f4b9ec872170ae241d6, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.167553 26669 heartbeater.cc:499] Master 127.24.187.254:37091 was elected leader, sending a full tablet report...
I20260319 16:32:56.167559 26811 heartbeater.cc:499] Master 127.24.187.254:37091 was elected leader, sending a full tablet report...
I20260319 16:32:56.167646 26978 consensus_queue.cc:260] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.167898 26953 heartbeater.cc:499] Master 127.24.187.254:37091 was elected leader, sending a full tablet report...
I20260319 16:32:56.167889 26978 ts_tablet_manager.cc:1434] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260319 16:32:56.202773 26812 tablet.cc:2379] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:32:56.297958 26670 tablet.cc:2379] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:32:56.343204 26984 raft_consensus.cc:493] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:32:56.343327 26984 raft_consensus.cc:515] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.343683 26984 leader_election.cc:290] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f00f007fbbe4b55a087b6eb4253f334 (127.24.187.194:46485), 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033)
I20260319 16:32:56.345916 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.343875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:37033 (local address 127.24.187.195:55903)
0319 16:32:56.343925 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.343937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.343977 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.344139 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.344140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.344142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:56.344348 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.344349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.345058 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.345059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.345737 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.345740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.345799 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.345803 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.345822 (+ 19us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.345852 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260319 16:32:56.346021 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.343886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46485 (local address 127.24.187.195:52591)
0319 16:32:56.344047 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.344055 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.344118 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.344247 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.344248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.344249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:56.344461 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.344463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.345156 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.345157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.345801 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.345804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.345873 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.345877 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.345905 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.345941 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260319 16:32:56.346367 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.343949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:55903 (local address 127.24.187.193:37033)
0319 16:32:56.344016 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.344018 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.344029 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:32:56.344059 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.344061 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.344073 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.344129 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:56.344431 (+ 302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.344988 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.345820 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.346251 (+ 431us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.346261 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.346282 (+ 21us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.346310 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:32:56.346385 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.343940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:52591 (local address 127.24.187.194:46485)
0319 16:32:56.344007 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.344009 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.344117 (+ 108us) server_negotiation.cc:408] Connection header received
0319 16:32:56.344178 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.344179 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.344190 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.344237 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:56.344533 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.345082 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.345882 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.346292 (+ 410us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.346300 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.346319 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.346340 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260319 16:32:56.346638 26623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d046ab17fc84c7f8601f7f52d211a23" candidate_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" is_pre_election: true
I20260319 16:32:56.346647 26765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d046ab17fc84c7f8601f7f52d211a23" candidate_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f00f007fbbe4b55a087b6eb4253f334" is_pre_election: true
I20260319 16:32:56.346757 26765 raft_consensus.cc:2468] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7136f2d91c7d4f4b9ec872170ae241d6 in term 0.
I20260319 16:32:56.346766 26623 raft_consensus.cc:2468] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7136f2d91c7d4f4b9ec872170ae241d6 in term 0.
I20260319 16:32:56.346962 26840 leader_election.cc:304] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7136f2d91c7d4f4b9ec872170ae241d6, 9b9fcfdd4e7e4c2e8a09e6326e9ed12a; no voters:
I20260319 16:32:56.347133 26984 raft_consensus.cc:2804] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:32:56.347185 26984 raft_consensus.cc:493] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:32:56.347209 26984 raft_consensus.cc:3060] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.347867 26984 raft_consensus.cc:515] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.347994 26984 leader_election.cc:290] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [CANDIDATE]: Term 1 election: Requested vote from peers 5f00f007fbbe4b55a087b6eb4253f334 (127.24.187.194:46485), 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033)
I20260319 16:32:56.348163 26765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d046ab17fc84c7f8601f7f52d211a23" candidate_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f00f007fbbe4b55a087b6eb4253f334"
I20260319 16:32:56.348179 26623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d046ab17fc84c7f8601f7f52d211a23" candidate_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a"
I20260319 16:32:56.348238 26623 raft_consensus.cc:3060] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.348233 26765 raft_consensus.cc:3060] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.348943 26765 raft_consensus.cc:2468] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7136f2d91c7d4f4b9ec872170ae241d6 in term 1.
I20260319 16:32:56.349038 26623 raft_consensus.cc:2468] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7136f2d91c7d4f4b9ec872170ae241d6 in term 1.
I20260319 16:32:56.349149 26839 leader_election.cc:304] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f00f007fbbe4b55a087b6eb4253f334, 7136f2d91c7d4f4b9ec872170ae241d6; no voters:
I20260319 16:32:56.349256 26984 raft_consensus.cc:2804] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:32:56.349431 26984 raft_consensus.cc:697] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 LEADER]: Becoming Leader. State: Replica: 7136f2d91c7d4f4b9ec872170ae241d6, State: Running, Role: LEADER
I20260319 16:32:56.349510 26984 consensus_queue.cc:237] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.350306 26463 catalog_manager.cc:5671] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195). New cstate: current_term: 1 leader_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } health_report { overall_health: HEALTHY } } }
W20260319 16:32:56.372517 26954 tablet.cc:2379] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:32:56.389312 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.387341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.24.187.195:39987)
0319 16:32:56.387404 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.387406 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.387414 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:32:56.387448 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.387449 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.387464 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.387512 (+ 48us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:32:56.387735 (+ 223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.388203 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.388826 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.388968 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.389002 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:32:56.389086 (+ 84us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:32:56.389148 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.389226 (+ 78us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.389250 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:32:56.390949 26765 raft_consensus.cc:1275] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Refusing update from remote peer 7136f2d91c7d4f4b9ec872170ae241d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:32:56.390949 26623 raft_consensus.cc:1275] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Refusing update from remote peer 7136f2d91c7d4f4b9ec872170ae241d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:32:56.391235 26984 consensus_queue.cc:1048] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:32:56.391366 26986 consensus_queue.cc:1048] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:32:56.394189 26990 mvcc.cc:204] Tried to move back new op lower bound from 7266049951294828544 to 7266049951128502272. Current Snapshot: MvccSnapshot[applied={T|T < 7266049951294828544}]
I20260319 16:32:56.396685 26463 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:32:56.396778 26463 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:32:56.397553 26463 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28: Sending DeleteTablet for 3 replicas of tablet 1d046ab17fc84c7f8601f7f52d211a23
I20260319 16:32:56.397873 26745 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d046ab17fc84c7f8601f7f52d211a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:32:56 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:35788
I20260319 16:32:56.397931 26603 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d046ab17fc84c7f8601f7f52d211a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:32:56 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:60034
I20260319 16:32:56.398110 26887 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d046ab17fc84c7f8601f7f52d211a23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:32:56 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:57502
I20260319 16:32:56.398108 26463 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:32:56.398233 26463 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:32:56.398322 26999 tablet_replica.cc:333] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: stopping tablet replica
I20260319 16:32:56.398396 26999 raft_consensus.cc:2243] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:32:56.398481 26999 raft_consensus.cc:2272] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:32:56.398581 27001 tablet_replica.cc:333] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: stopping tablet replica
I20260319 16:32:56.398593 27000 tablet_replica.cc:333] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: stopping tablet replica
I20260319 16:32:56.398661 27000 raft_consensus.cc:2243] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:32:56.398716 27001 raft_consensus.cc:2243] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:32:56.399077 27001 raft_consensus.cc:2272] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:32:56.399165 27000 raft_consensus.cc:2272] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:32:56.399411 26603 tablet_service.cc:1511] Processing CreateTablet for tablet 3e42bfb461ed4b899793981b838b186f (DEFAULT_TABLE table=test-table [id=e0aeda19cd7347f98f20a611402f4b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:56.399526 26603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e42bfb461ed4b899793981b838b186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.400518 26977 tablet_bootstrap.cc:492] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Bootstrap starting.
I20260319 16:32:56.399488 26887 tablet_service.cc:1511] Processing CreateTablet for tablet 3e42bfb461ed4b899793981b838b186f (DEFAULT_TABLE table=test-table [id=e0aeda19cd7347f98f20a611402f4b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:56.400892 26887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e42bfb461ed4b899793981b838b186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.400934 26977 tablet_bootstrap.cc:654] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.401507 26999 ts_tablet_manager.cc:1916] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:32:56.401772 27000 ts_tablet_manager.cc:1916] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:32:56.402138 26978 tablet_bootstrap.cc:492] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: Bootstrap starting.
I20260319 16:32:56.402434 26745 tablet_service.cc:1511] Processing CreateTablet for tablet 3e42bfb461ed4b899793981b838b186f (DEFAULT_TABLE table=test-table [id=e0aeda19cd7347f98f20a611402f4b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:32:56.402520 26745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e42bfb461ed4b899793981b838b186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.402547 26978 tablet_bootstrap.cc:654] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.403455 26999 ts_tablet_manager.cc:1929] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:32:56.403558 26999 log.cc:1199] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/1d046ab17fc84c7f8601f7f52d211a23
I20260319 16:32:56.403702 26976 tablet_bootstrap.cc:492] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: Bootstrap starting.
I20260319 16:32:56.403816 26999 ts_tablet_manager.cc:1950] T 1d046ab17fc84c7f8601f7f52d211a23 P 5f00f007fbbe4b55a087b6eb4253f334: Deleting consensus metadata
I20260319 16:32:56.404049 26976 tablet_bootstrap.cc:654] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.404139 27000 ts_tablet_manager.cc:1929] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:32:56.404193 27000 log.cc:1199] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/1d046ab17fc84c7f8601f7f52d211a23
I20260319 16:32:56.404495 27000 ts_tablet_manager.cc:1950] T 1d046ab17fc84c7f8601f7f52d211a23 P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Deleting consensus metadata
I20260319 16:32:56.404580 26449 catalog_manager.cc:5002] TS 5f00f007fbbe4b55a087b6eb4253f334 (127.24.187.194:46485): tablet 1d046ab17fc84c7f8601f7f52d211a23 (table test-table [id=67b163a4918f4729adf30687a3e13323]) successfully deleted
I20260319 16:32:56.404708 26976 tablet_bootstrap.cc:492] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: No bootstrap required, opened a new log
I20260319 16:32:56.404767 26976 ts_tablet_manager.cc:1403] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:32:56.404995 26976 raft_consensus.cc:359] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.405058 26976 raft_consensus.cc:385] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.405063 26977 tablet_bootstrap.cc:492] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: No bootstrap required, opened a new log
I20260319 16:32:56.405077 26976 raft_consensus.cc:740] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f00f007fbbe4b55a087b6eb4253f334, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.405098 26977 ts_tablet_manager.cc:1403] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260319 16:32:56.405118 26976 consensus_queue.cc:260] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.405193 26976 ts_tablet_manager.cc:1434] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:56.405171 26450 catalog_manager.cc:5002] TS 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033): tablet 1d046ab17fc84c7f8601f7f52d211a23 (table test-table [id=67b163a4918f4729adf30687a3e13323]) successfully deleted
I20260319 16:32:56.405226 26977 raft_consensus.cc:359] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.405277 26977 raft_consensus.cc:385] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.405293 26977 raft_consensus.cc:740] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9fcfdd4e7e4c2e8a09e6326e9ed12a, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.405339 26977 consensus_queue.cc:260] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.405422 26977 ts_tablet_manager.cc:1434] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:56.406119 26978 tablet_bootstrap.cc:492] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: No bootstrap required, opened a new log
I20260319 16:32:56.406163 26978 ts_tablet_manager.cc:1403] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260319 16:32:56.406273 27001 ts_tablet_manager.cc:1916] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:32:56.406299 26978 raft_consensus.cc:359] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.406340 26978 raft_consensus.cc:385] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.406351 26978 raft_consensus.cc:740] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7136f2d91c7d4f4b9ec872170ae241d6, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.406392 26978 consensus_queue.cc:260] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.406483 26978 ts_tablet_manager.cc:1434] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:56.407820 27001 ts_tablet_manager.cc:1929] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:32:56.407886 27001 log.cc:1199] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/1d046ab17fc84c7f8601f7f52d211a23
I20260319 16:32:56.408128 27001 ts_tablet_manager.cc:1950] T 1d046ab17fc84c7f8601f7f52d211a23 P 7136f2d91c7d4f4b9ec872170ae241d6: Deleting consensus metadata
I20260319 16:32:56.408563 26450 catalog_manager.cc:5002] TS 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195:39987): tablet 1d046ab17fc84c7f8601f7f52d211a23 (table test-table [id=67b163a4918f4729adf30687a3e13323]) successfully deleted
I20260319 16:32:56.546284 26982 raft_consensus.cc:493] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:32:56.546386 26982 raft_consensus.cc:515] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.546711 26982 leader_election.cc:290] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033), 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195:39987)
I20260319 16:32:56.548977 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.546865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:37033 (local address 127.24.187.194:51773)
0319 16:32:56.546974 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.546986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.547048 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.547362 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.547363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.547365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:56.547547 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.547548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.548138 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.548139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.548749 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.548751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.548817 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.548820 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.548848 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.548883 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260319 16:32:56.548980 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.547035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:39987 (local address 127.24.187.194:53167)
0319 16:32:56.547140 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.547151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.547221 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.547324 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.547325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.547328 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:32:56.547526 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.547528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.548140 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.548141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.548741 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.548743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.548803 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.548809 (+ 6us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.548838 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.548877 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260319 16:32:56.549352 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.547118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:53167 (local address 127.24.187.195:39987)
0319 16:32:56.547170 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.547172 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.547205 (+ 33us) server_negotiation.cc:408] Connection header received
0319 16:32:56.547277 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.547278 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.547285 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.547319 (+ 34us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:56.547609 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.548068 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.548823 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.549232 (+ 409us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.549242 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.549268 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.549305 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:32:56.549362 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.546945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:51773 (local address 127.24.187.193:37033)
0319 16:32:56.547000 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.547002 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.547035 (+ 33us) server_negotiation.cc:408] Connection header received
0319 16:32:56.547176 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.547178 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.547190 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.547252 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:32:56.547632 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.548069 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.548832 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.549242 (+ 410us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.549251 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.549274 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.549303 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260319 16:32:56.549656 26623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e42bfb461ed4b899793981b838b186f" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" is_pre_election: true
I20260319 16:32:56.549654 26907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e42bfb461ed4b899793981b838b186f" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" is_pre_election: true
I20260319 16:32:56.549767 26623 raft_consensus.cc:2468] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 0.
I20260319 16:32:56.549780 26907 raft_consensus.cc:2468] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 0.
I20260319 16:32:56.549919 26698 leader_election.cc:304] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f00f007fbbe4b55a087b6eb4253f334, 9b9fcfdd4e7e4c2e8a09e6326e9ed12a; no voters:
I20260319 16:32:56.550040 26982 raft_consensus.cc:2804] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:32:56.550082 26982 raft_consensus.cc:493] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:32:56.550108 26982 raft_consensus.cc:3060] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.550727 26982 raft_consensus.cc:515] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.550836 26982 leader_election.cc:290] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 election: Requested vote from peers 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033), 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195:39987)
I20260319 16:32:56.550993 26623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e42bfb461ed4b899793981b838b186f" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a"
I20260319 16:32:56.551023 26907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e42bfb461ed4b899793981b838b186f" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7136f2d91c7d4f4b9ec872170ae241d6"
I20260319 16:32:56.551064 26623 raft_consensus.cc:3060] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.551079 26907 raft_consensus.cc:3060] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.551657 26623 raft_consensus.cc:2468] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 1.
I20260319 16:32:56.551679 26907 raft_consensus.cc:2468] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 1.
I20260319 16:32:56.551827 26698 leader_election.cc:304] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f00f007fbbe4b55a087b6eb4253f334, 9b9fcfdd4e7e4c2e8a09e6326e9ed12a; no voters:
I20260319 16:32:56.551923 26982 raft_consensus.cc:2804] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:32:56.552038 26982 raft_consensus.cc:697] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 LEADER]: Becoming Leader. State: Replica: 5f00f007fbbe4b55a087b6eb4253f334, State: Running, Role: LEADER
I20260319 16:32:56.552107 26982 consensus_queue.cc:237] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.552748 26464 catalog_manager.cc:5671] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f00f007fbbe4b55a087b6eb4253f334 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } health_report { overall_health: UNKNOWN } } }
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937975, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:32:56.612761 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.606488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37091 (local address 127.0.0.1:37598)
0319 16:32:56.606711 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.606720 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.606780 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.606935 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.606936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.607250 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:32:56.607411 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.607414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.608081 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.608083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.608516 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.608518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.608575 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.609057 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:32:56.609061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:32:56.610473 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:32:56.611826 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:56.611832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:56.611834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:56.612017 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:56.612152 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:32:56.612153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:32:56.612153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:32:56.612178 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:32:56.612473 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:32:56.612478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:32:56.612524 (+ 46us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.612592 (+ 68us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.612687 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":72,"threads_started":1}
I20260319 16:32:56.612825 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.606617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.24.187.254:37091)
0319 16:32:56.606677 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.606679 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.606786 (+ 107us) server_negotiation.cc:408] Connection header received
0319 16:32:56.606853 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.606855 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.606867 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.606921 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:56.607519 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.608013 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.608631 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.608785 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.610564 (+ 1779us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:56.610569 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:56.610570 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:56.610576 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:56.611744 (+ 1168us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.612083 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.612084 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.612084 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.612101 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.612260 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.612261 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.612262 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.612385 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:56.612450 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.612616 (+ 166us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.612746 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":25}
I20260319 16:32:56.614544 26464 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:37598:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260319 16:32:56.614724 26464 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:32:56.616078 26887 tablet_service.cc:1511] Processing CreateTablet for tablet ce7abae385c8403ab8c6c8ac1301ba2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5e098020cdf4f41bd6468a1309a7774]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:32:56.616065 26603 tablet_service.cc:1511] Processing CreateTablet for tablet ce7abae385c8403ab8c6c8ac1301ba2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5e098020cdf4f41bd6468a1309a7774]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:32:56.616093 26745 tablet_service.cc:1511] Processing CreateTablet for tablet ce7abae385c8403ab8c6c8ac1301ba2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5e098020cdf4f41bd6468a1309a7774]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:32:56.616214 26603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7abae385c8403ab8c6c8ac1301ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.616212 26887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7abae385c8403ab8c6c8ac1301ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.616215 26745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7abae385c8403ab8c6c8ac1301ba2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:32:56.617193 26976 tablet_bootstrap.cc:492] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: Bootstrap starting.
I20260319 16:32:56.617548 26978 tablet_bootstrap.cc:492] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: Bootstrap starting.
I20260319 16:32:56.617549 26977 tablet_bootstrap.cc:492] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Bootstrap starting.
I20260319 16:32:56.617584 26976 tablet_bootstrap.cc:654] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.617897 26978 tablet_bootstrap.cc:654] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.617964 26977 tablet_bootstrap.cc:654] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Neither blocks nor log segments found. Creating new log.
I20260319 16:32:56.618327 26976 tablet_bootstrap.cc:492] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: No bootstrap required, opened a new log
I20260319 16:32:56.618372 26976 ts_tablet_manager.cc:1403] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:32:56.618407 26978 tablet_bootstrap.cc:492] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: No bootstrap required, opened a new log
I20260319 16:32:56.618448 26978 ts_tablet_manager.cc:1403] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:32:56.618527 26976 raft_consensus.cc:359] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.618557 26978 raft_consensus.cc:359] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.618599 26976 raft_consensus.cc:385] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.618611 26978 raft_consensus.cc:385] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.618620 26976 raft_consensus.cc:740] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f00f007fbbe4b55a087b6eb4253f334, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.618628 26978 raft_consensus.cc:740] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7136f2d91c7d4f4b9ec872170ae241d6, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.618665 26976 consensus_queue.cc:260] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.618682 26978 consensus_queue.cc:260] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.618747 26976 ts_tablet_manager.cc:1434] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260319 16:32:56.618759 26978 ts_tablet_manager.cc:1434] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:32:56.618835 26982 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.618877 26982 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:32:56.618877 26977 tablet_bootstrap.cc:492] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: No bootstrap required, opened a new log
I20260319 16:32:56.618847 26996 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.618906 26996 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:32:56.618914 26977 ts_tablet_manager.cc:1403] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:32:56.619056 26977 raft_consensus.cc:359] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.619107 26977 raft_consensus.cc:385] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:32:56.619124 26977 raft_consensus.cc:740] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9fcfdd4e7e4c2e8a09e6326e9ed12a, State: Initialized, Role: FOLLOWER
I20260319 16:32:56.619160 26977 consensus_queue.cc:260] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.619236 26977 ts_tablet_manager.cc:1434] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260319 16:32:56.619359 26983 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.619434 26983 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:32:56.639910 26982 raft_consensus.cc:493] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:32:56.640007 26982 raft_consensus.cc:515] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.640156 26982 leader_election.cc:290] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033), 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195:39987)
I20260319 16:32:56.640378 26623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce7abae385c8403ab8c6c8ac1301ba2e" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" is_pre_election: true
I20260319 16:32:56.640409 26907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce7abae385c8403ab8c6c8ac1301ba2e" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" is_pre_election: true
I20260319 16:32:56.640487 26907 raft_consensus.cc:2468] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 0.
I20260319 16:32:56.640487 26623 raft_consensus.cc:2468] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 0.
I20260319 16:32:56.640650 26698 leader_election.cc:304] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f00f007fbbe4b55a087b6eb4253f334, 9b9fcfdd4e7e4c2e8a09e6326e9ed12a; no voters:
I20260319 16:32:56.640782 26982 raft_consensus.cc:2804] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:32:56.640839 26982 raft_consensus.cc:493] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:32:56.640861 26982 raft_consensus.cc:3060] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.641485 26982 raft_consensus.cc:515] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.641602 26982 leader_election.cc:290] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 election: Requested vote from peers 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033), 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195:39987)
I20260319 16:32:56.641849 26623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce7abae385c8403ab8c6c8ac1301ba2e" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a"
I20260319 16:32:56.641853 26907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce7abae385c8403ab8c6c8ac1301ba2e" candidate_uuid: "5f00f007fbbe4b55a087b6eb4253f334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7136f2d91c7d4f4b9ec872170ae241d6"
I20260319 16:32:56.641921 26907 raft_consensus.cc:3060] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.641923 26623 raft_consensus.cc:3060] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:32:56.642485 26623 raft_consensus.cc:2468] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 1.
I20260319 16:32:56.642510 26907 raft_consensus.cc:2468] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f00f007fbbe4b55a087b6eb4253f334 in term 1.
I20260319 16:32:56.642652 26698 leader_election.cc:304] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f00f007fbbe4b55a087b6eb4253f334, 7136f2d91c7d4f4b9ec872170ae241d6; no voters:
I20260319 16:32:56.642740 26982 raft_consensus.cc:2804] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:32:56.642792 26982 raft_consensus.cc:697] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 LEADER]: Becoming Leader. State: Replica: 5f00f007fbbe4b55a087b6eb4253f334, State: Running, Role: LEADER
I20260319 16:32:56.642833 26982 consensus_queue.cc:237] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } }
I20260319 16:32:56.643191 27007 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: TxnStatusTablet state changed. Reason: New leader 5f00f007fbbe4b55a087b6eb4253f334. Latest consensus state: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.643291 27007 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:32:56.643481 27019 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260319 16:32:56.643584 27019 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260319 16:32:56.643540 26463 catalog_manager.cc:5671] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f00f007fbbe4b55a087b6eb4253f334 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } health_report { overall_health: UNKNOWN } } }
I20260319 16:32:56.650501 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.648148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46485 (local address 127.0.0.1:41196)
0319 16:32:56.648278 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.648289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.648344 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.648485 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.648486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.648487 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:32:56.648630 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.648631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.649499 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.649500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.649931 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.649933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.649986 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.650002 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:32:56.650381 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:32:56.650385 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.650407 (+ 22us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.650446 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260319 16:32:56.650503 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.648201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.24.187.194:46485)
0319 16:32:56.648254 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.648256 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.648336 (+ 80us) server_negotiation.cc:408] Connection header received
0319 16:32:56.648398 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.648399 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.648417 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.648484 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:32:56.648683 (+ 199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.649431 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.649999 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.650184 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.650218 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:32:56.650318 (+ 100us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:32:56.650394 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.650412 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.650434 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:32:56.651558 26623 raft_consensus.cc:1275] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Refusing update from remote peer 5f00f007fbbe4b55a087b6eb4253f334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:32:56.651597 26907 raft_consensus.cc:1275] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Refusing update from remote peer 5f00f007fbbe4b55a087b6eb4253f334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:32:56.651885 26982 consensus_queue.cc:1048] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:32:56.651997 27007 consensus_queue.cc:1048] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:32:56.652594 26983 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: TxnStatusTablet state changed. Reason: New leader 5f00f007fbbe4b55a087b6eb4253f334. Latest consensus state: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.652621 26996 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: TxnStatusTablet state changed. Reason: New leader 5f00f007fbbe4b55a087b6eb4253f334. Latest consensus state: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.652689 26983 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:32:56.652696 26996 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:32:56.653546 27007 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.653623 27007 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:32:56.654309 26982 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.654385 26982 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 5f00f007fbbe4b55a087b6eb4253f334: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:32:56.654538 26983 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.654610 26983 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:32:56.655211 26996 tablet_replica.cc:442] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f00f007fbbe4b55a087b6eb4253f334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 } } peers { permanent_uuid: "5f00f007fbbe4b55a087b6eb4253f334" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46485 } } peers { permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 } } }
I20260319 16:32:56.655292 26996 tablet_replica.cc:445] T ce7abae385c8403ab8c6c8ac1301ba2e P 7136f2d91c7d4f4b9ec872170ae241d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:32:56.657838 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.655754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.24.187.194:46485)
0319 16:32:56.655865 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.655867 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.655875 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:32:56.655892 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.655893 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.655904 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.655966 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:32:56.656160 (+ 194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.656864 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.657380 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.657567 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.657588 (+ 21us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:32:56.657614 (+ 26us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:32:56.657677 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.657753 (+ 76us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.657774 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260319 16:32:56.662218 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.660022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:50213 (local address 127.24.187.194:46485)
0319 16:32:56.660087 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.660089 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.660147 (+ 58us) server_negotiation.cc:408] Connection header received
0319 16:32:56.660181 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.660181 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.660190 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.660254 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:32:56.660543 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.661221 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.661821 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.661984 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.662002 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:32:56.662026 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:32:56.662084 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.662118 (+ 34us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.662135 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260319 16:32:56.662235 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.659958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46485 (local address 127.24.187.194:50213)
0319 16:32:56.660085 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0319 16:32:56.660094 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:32:56.660144 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:32:56.660259 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:32:56.660260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:32:56.660262 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:32:56.660486 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.660488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.661294 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.661295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:32:56.661753 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:32:56.661756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.661811 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.661826 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:32:56.662084 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:32:56.662085 (+ 1us) client_negotiation.cc:769] Sending connection context
0319 16:32:56.662111 (+ 26us) client_negotiation.cc:240] Negotiation successful
0319 16:32:56.662143 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
I20260319 16:32:56.664168 26623 raft_consensus.cc:1275] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Refusing update from remote peer 5f00f007fbbe4b55a087b6eb4253f334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:32:56.664166 26906 raft_consensus.cc:1275] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Refusing update from remote peer 5f00f007fbbe4b55a087b6eb4253f334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:32:56.664389 27007 consensus_queue.cc:1048] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9fcfdd4e7e4c2e8a09e6326e9ed12a" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:32:56.664487 26982 consensus_queue.cc:1048] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7136f2d91c7d4f4b9ec872170ae241d6" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:32:56.665843 26991 mvcc.cc:204] Tried to move back new op lower bound from 7266049952415158272 to 7266049951957958656. Current Snapshot: MvccSnapshot[applied={T|T < 7266049952415158272}]
I20260319 16:32:56.666261 26989 mvcc.cc:204] Tried to move back new op lower bound from 7266049952415158272 to 7266049951957958656. Current Snapshot: MvccSnapshot[applied={T|T < 7266049952415158272}]
I20260319 16:32:56.676292 26463 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:32:56.676405 26463 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:32:56.677006 26463 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 48e2e5682f83453ca7e90b48d3255f28: Sending DeleteTablet for 3 replicas of tablet 3e42bfb461ed4b899793981b838b186f
I20260319 16:32:56.677335 26603 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e42bfb461ed4b899793981b838b186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:32:56 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:60034
I20260319 16:32:56.677367 26745 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e42bfb461ed4b899793981b838b186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:32:56 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:35788
I20260319 16:32:56.677493 27000 tablet_replica.cc:333] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: stopping tablet replica
I20260319 16:32:56.677541 27000 raft_consensus.cc:2243] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:32:56.677573 27000 raft_consensus.cc:2272] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:32:56.677784 26887 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e42bfb461ed4b899793981b838b186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:32:56 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:57502
I20260319 16:32:56.677865 27000 ts_tablet_manager.cc:1916] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:32:56.677891 26999 tablet_replica.cc:333] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: stopping tablet replica
I20260319 16:32:56.677901 27001 tablet_replica.cc:333] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: stopping tablet replica
I20260319 16:32:56.677942 26999 raft_consensus.cc:2243] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:32:56.677949 27001 raft_consensus.cc:2243] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:32:56.677982 27001 raft_consensus.cc:2272] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:32:56.678017 26999 raft_consensus.cc:2272] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:32:56.678359 26999 ts_tablet_manager.cc:1916] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:32:56.679256 27001 ts_tablet_manager.cc:1916] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:32:56.679355 27000 ts_tablet_manager.cc:1929] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:32:56.679397 27000 log.cc:1199] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/3e42bfb461ed4b899793981b838b186f
I20260319 16:32:56.679612 27000 ts_tablet_manager.cc:1950] T 3e42bfb461ed4b899793981b838b186f P 9b9fcfdd4e7e4c2e8a09e6326e9ed12a: Deleting consensus metadata
I20260319 16:32:56.679805 26999 ts_tablet_manager.cc:1929] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:32:56.679850 26999 log.cc:1199] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/3e42bfb461ed4b899793981b838b186f
I20260319 16:32:56.679906 26450 catalog_manager.cc:5002] TS 9b9fcfdd4e7e4c2e8a09e6326e9ed12a (127.24.187.193:37033): tablet 3e42bfb461ed4b899793981b838b186f (table test-table [id=e0aeda19cd7347f98f20a611402f4b78]) successfully deleted
I20260319 16:32:56.680045 26999 ts_tablet_manager.cc:1950] T 3e42bfb461ed4b899793981b838b186f P 5f00f007fbbe4b55a087b6eb4253f334: Deleting consensus metadata
I20260319 16:32:56.680459 26449 catalog_manager.cc:5002] TS 5f00f007fbbe4b55a087b6eb4253f334 (127.24.187.194:46485): tablet 3e42bfb461ed4b899793981b838b186f (table test-table [id=e0aeda19cd7347f98f20a611402f4b78]) successfully deleted
I20260319 16:32:56.680656 27001 ts_tablet_manager.cc:1929] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:32:56.680714 27001 log.cc:1199] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/3e42bfb461ed4b899793981b838b186f
I20260319 16:32:56.680898 27001 ts_tablet_manager.cc:1950] T 3e42bfb461ed4b899793981b838b186f P 7136f2d91c7d4f4b9ec872170ae241d6: Deleting consensus metadata
I20260319 16:32:56.681324 26450 catalog_manager.cc:5002] TS 7136f2d91c7d4f4b9ec872170ae241d6 (127.24.187.195:39987): tablet 3e42bfb461ed4b899793981b838b186f (table test-table [id=e0aeda19cd7347f98f20a611402f4b78]) successfully deleted
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[26416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937976, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:32:56.689144 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.680404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.24.187.193:37033)
0319 16:32:56.680447 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.680449 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.681879 (+ 1430us) server_negotiation.cc:408] Connection header received
0319 16:32:56.681912 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.681913 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.681924 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.681984 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:56.682635 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.683403 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.684038 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.684209 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.686049 (+ 1840us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:56.686060 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:56.686060 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:56.686068 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:56.688039 (+ 1971us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.688426 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.688430 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.688430 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.688477 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.688593 (+ 116us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.688594 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.688594 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.688712 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:56.688856 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.688950 (+ 94us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.689045 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260319 16:32:56.689500 26663 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39466
I20260319 16:32:56.695900 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:32:56.690507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.24.187.254:37091)
0319 16:32:56.690622 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0319 16:32:56.690624 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:32:56.690634 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:32:56.690692 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:32:56.690693 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:32:56.690704 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:32:56.690746 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:32:56.691331 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.691810 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:32:56.692302 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:32:56.692465 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:32:56.693366 (+ 901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:32:56.693370 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:32:56.693371 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:32:56.693378 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:32:56.694848 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.695128 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.695129 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.695129 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.695153 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:32:56.695276 (+ 123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:32:56.695277 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:32:56.695278 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:32:56.695493 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:32:56.695608 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0319 16:32:56.695734 (+ 126us) server_negotiation.cc:300] Negotiation successful
0319 16:32:56.695812 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260319 16:32:56.696110 26463 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35656
I20260319 16:32:56.696800 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 26538
I20260319 16:32:56.702358 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 26680
I20260319 16:32:56.707204 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 26822
I20260319 16:32:56.711871 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 26432
2026-03-19T16:32:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3209 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[27038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[27038](info): set up 2 sockets
Mar 19 16:32:56 dist-test-slave-svn8 krb5kdc[27038](info): commencing operation
krb5kdc: starting...
W20260319 16:32:58.645696 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.914s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260319 16:32:58.973157 25327 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.327s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260319 16:32:59.260237 25327 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.287s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260319 16:32:59.559401 25327 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.299s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/test-user.keytab.
W20260319 16:33:00.629217 25327 mini_kdc.cc:290] Time spent creating keytab for test-user: real 1.070s user 0.001s sys 0.001s
Mar 19 16:33:00 dist-test-slave-svn8 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
W20260319 16:33:00.820091 25327 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.24.187.254: real 0.171s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:00.842139 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35717
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:00.919600 27054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:00.919801 27054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:00.919823 27054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:00.921592 27054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:00.921654 27054 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260319 16:33:00.921689 27054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:00.921702 27054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:00.921738 27054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:00.921754 27054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:00.923537 27054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:35717
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:00.923776 27054 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:00.923956 27054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:00.926231 27060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:00.926221 27059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:00.926224 27062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:00.926529 27054 server_base.cc:1060] running on GCE node
I20260319 16:33:00.926720 27054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:00.926951 27054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:00.928113 27054 hybrid_clock.cc:648] HybridClock initialized: now 1773937980928091 us; error 37 us; skew 500 ppm
Mar 19 16:33:00 dist-test-slave-svn8 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937980, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:00.930510 27054 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:00.931082 27054 webserver.cc:492] Webserver started at http://127.24.187.254:36763/ using document root <none> and password file <none>
I20260319 16:33:00.931253 27054 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:00.931289 27054 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:00.931392 27054 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:00.932417 27054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "91df8ebd2d4f4aaca509fd688adf73fb"
format_stamp: "Formatted at 2026-03-19 16:33:00 on dist-test-slave-svn8"
server_key: "2460c33f1bcb8dac920dcac95f5e7dbd"
server_key_iv: "5b16bf6911f0fb873860edb083cf330e"
server_key_version: "encryptionkey@0"
I20260319 16:33:00.932729 27054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "91df8ebd2d4f4aaca509fd688adf73fb"
format_stamp: "Formatted at 2026-03-19 16:33:00 on dist-test-slave-svn8"
server_key: "2460c33f1bcb8dac920dcac95f5e7dbd"
server_key_iv: "5b16bf6911f0fb873860edb083cf330e"
server_key_version: "encryptionkey@0"
I20260319 16:33:00.933970 27054 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:33:00.934811 27069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:00.935034 27054 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:00.935101 27054 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "91df8ebd2d4f4aaca509fd688adf73fb"
format_stamp: "Formatted at 2026-03-19 16:33:00 on dist-test-slave-svn8"
server_key: "2460c33f1bcb8dac920dcac95f5e7dbd"
server_key_iv: "5b16bf6911f0fb873860edb083cf330e"
server_key_version: "encryptionkey@0"
I20260319 16:33:00.935169 27054 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:00.966627 27054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:00.976694 27054 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:00.976832 27054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:00.981493 27054 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:35717
I20260319 16:33:00.981536 27131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:35717 every 8 connection(s)
I20260319 16:33:00.981884 27054 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:00.982470 27132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:00.984936 27132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Bootstrap starting.
I20260319 16:33:00.985513 27132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:00.985785 27132 log.cc:826] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:00.986543 27132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: No bootstrap required, opened a new log
I20260319 16:33:00.987267 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 27054
I20260319 16:33:00.987363 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:00.987565 25327 external_mini_cluster.cc:1468] Setting key 0e4ae91531e1a786b827e0e375745797
I20260319 16:33:00.988293 27132 raft_consensus.cc:359] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35717 } }
I20260319 16:33:00.988420 27132 raft_consensus.cc:385] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:00.988440 27132 raft_consensus.cc:740] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91df8ebd2d4f4aaca509fd688adf73fb, State: Initialized, Role: FOLLOWER
I20260319 16:33:00.988521 27132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35717 } }
I20260319 16:33:00.988579 27132 raft_consensus.cc:399] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:00.988600 27132 raft_consensus.cc:493] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:00.988651 27132 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:00.989347 27132 raft_consensus.cc:515] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35717 } }
I20260319 16:33:00.989462 27132 leader_election.cc:304] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91df8ebd2d4f4aaca509fd688adf73fb; no voters:
I20260319 16:33:00.989606 27132 leader_election.cc:290] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:00.989687 27137 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:00.989899 27132 sys_catalog.cc:565] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:00.989909 27137 raft_consensus.cc:697] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [term 1 LEADER]: Becoming Leader. State: Replica: 91df8ebd2d4f4aaca509fd688adf73fb, State: Running, Role: LEADER
I20260319 16:33:00.990069 27137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35717 } }
I20260319 16:33:00.990386 27139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91df8ebd2d4f4aaca509fd688adf73fb. Latest consensus state: current_term: 1 leader_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35717 } } }
I20260319 16:33:00.990520 27139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:00.990695 27138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91df8ebd2d4f4aaca509fd688adf73fb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35717 } } }
I20260319 16:33:00.991001 27138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:00.991080 27141 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:00.991568 27141 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:00.993085 27141 catalog_manager.cc:1357] Generated new cluster ID: e315f6c828ad48cfb3b5b00257335b03
I20260319 16:33:00.993131 27141 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:33:00 dist-test-slave-svn8 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:00.997845 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:00.988123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.24.187.254:35717)
0319 16:33:00.988311 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:00.988319 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:00.988336 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:33:00.988480 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:00.988486 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:00.988534 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:00.988688 (+ 154us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:00.989667 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:00.990847 (+ 1180us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:00.991800 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:00.992047 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:00.994243 (+ 2196us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:00.994248 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:00.994249 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:00.994267 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:00.996133 (+ 1866us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:00.996510 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:00.996513 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:00.996513 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:00.996582 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:00.996737 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:00.996737 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:00.996738 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:00.996968 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:00.997082 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:00.997520 (+ 438us) server_negotiation.cc:300] Negotiation successful
0319 16:33:00.997646 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260319 16:33:01.002822 27141 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:01.004122 27141 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:01.011454 27141 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Generated new TSK 0
I20260319 16:33:01.011672 27141 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:01.061429 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:35717
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:01.138017 27160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:01.138224 27160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:01.138262 27160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:01.139859 27160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:01.139914 27160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:01.139935 27160 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260319 16:33:01.139946 27160 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:33:01.139961 27160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:01.141667 27160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:01.141911 27160 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:01.142092 27160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:01.144431 27166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:01.144508 27165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:01.144646 27168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:01.144551 27160 server_base.cc:1060] running on GCE node
I20260319 16:33:01.144860 27160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:01.145056 27160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:01.146214 27160 hybrid_clock.cc:648] HybridClock initialized: now 1773937981146195 us; error 30 us; skew 500 ppm
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:01.148780 27160 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:01.149302 27160 webserver.cc:492] Webserver started at http://127.24.187.193:34481/ using document root <none> and password file <none>
I20260319 16:33:01.149482 27160 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:01.149518 27160 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:01.149643 27160 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:01.150565 27160 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "47d2e70f3f8d4ab0810c3db8f74e4327"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "18ef374c5434a5ec809d20449bccc7ee"
server_key_iv: "6d033a0260cbeb19befda0bc8cd1cdde"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.150874 27160 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "47d2e70f3f8d4ab0810c3db8f74e4327"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "18ef374c5434a5ec809d20449bccc7ee"
server_key_iv: "6d033a0260cbeb19befda0bc8cd1cdde"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.152048 27160 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:01.152751 27175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.152906 27160 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:01.152968 27160 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "47d2e70f3f8d4ab0810c3db8f74e4327"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "18ef374c5434a5ec809d20449bccc7ee"
server_key_iv: "6d033a0260cbeb19befda0bc8cd1cdde"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.153033 27160 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:01.169611 27160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:01.171715 27160 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:01.171849 27160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:01.172360 27160 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:01.172398 27160 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.172426 27160 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:01.172554 27160 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:01.179644 27160 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:33649
I20260319 16:33:01.179730 27290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:33649 every 8 connection(s)
I20260319 16:33:01.180016 27160 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:01.182214 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.172812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:36519 (local address 127.24.187.254:35717)
0319 16:33:01.172866 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.172869 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.173082 (+ 213us) server_negotiation.cc:408] Connection header received
0319 16:33:01.173607 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.173609 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.173626 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.173682 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.174727 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.175482 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.176267 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.176451 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.178974 (+ 2523us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.178981 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.178981 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.178991 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.180732 (+ 1741us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.181280 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.181281 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.181281 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.181307 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.181562 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.181563 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.181564 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.181693 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.181795 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.182055 (+ 260us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.182150 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:33:01.182274 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.172769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35717 (local address 127.24.187.193:36519)
0319 16:33:01.172993 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.173015 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.173521 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.173765 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.173770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.174142 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:01.174560 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.174578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.175585 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.175586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.176168 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.176171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.176311 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.176876 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:01.176901 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:01.178854 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:01.180834 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.180841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.180843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.181194 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.181426 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.181427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.181427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.181493 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.181789 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:01.181793 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:01.181905 (+ 112us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.182003 (+ 98us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.182124 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260319 16:33:01.182844 27291 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35717
I20260319 16:33:01.182921 27291 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:01.183104 27291 heartbeater.cc:507] Master 127.24.187.254:35717 requested a full tablet report, sending...
I20260319 16:33:01.183660 27085 ts_manager.cc:194] Registered new tserver with Master: 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649)
I20260319 16:33:01.184388 27085 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:36519
I20260319 16:33:01.186683 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 27160
I20260319 16:33:01.186772 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:01.186962 25327 external_mini_cluster.cc:1468] Setting key 32c51d667e1e8fc6aab70a6eb1e6edc4
I20260319 16:33:01.188885 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.184001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:52999 (local address 127.24.187.254:35717)
0319 16:33:01.184048 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.184051 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.184120 (+ 69us) server_negotiation.cc:408] Connection header received
0319 16:33:01.184191 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.184192 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.184202 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.184334 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.184821 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.185267 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.185803 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.185945 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.186581 (+ 636us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.186584 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.186584 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.186590 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.187988 (+ 1398us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.188334 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.188335 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.188335 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.188360 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.188501 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.188501 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.188501 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.188577 (+ 76us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.188623 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.188751 (+ 128us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.188823 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:33:01.188927 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.183941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35717 (local address 127.24.187.193:52999)
0319 16:33:01.184064 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.184074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.184134 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.184338 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.184340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.184507 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:01.184738 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.184740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.185331 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.185332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.185693 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.185695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.185773 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.186178 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:01.186182 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:01.186456 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:01.188078 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.188079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.188080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.188242 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.188421 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.188422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.188422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.188454 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.188628 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:01.188628 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:01.188672 (+ 44us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.188726 (+ 54us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.188827 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:01.237191 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:35717
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:01.317230 27302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:01.317425 27302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:01.317507 27302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:01.319053 27302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:01.319113 27302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:01.319140 27302 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260319 16:33:01.319155 27302 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:33:01.319176 27302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:01.320890 27302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:01.321128 27302 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:01.321311 27302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:01.323814 27310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:01.323807 27307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:01.323858 27308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:01.324005 27302 server_base.cc:1060] running on GCE node
I20260319 16:33:01.324211 27302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:01.324431 27302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:01.325559 27302 hybrid_clock.cc:648] HybridClock initialized: now 1773937981325546 us; error 27 us; skew 500 ppm
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:01.327934 27302 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:01.328543 27302 webserver.cc:492] Webserver started at http://127.24.187.194:42429/ using document root <none> and password file <none>
I20260319 16:33:01.328715 27302 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:01.328751 27302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:01.328853 27302 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:01.329785 27302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "7b815513b7f84c3fb82696f646ec8487"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "4bcf767b026bf1fff177c250a8e53fd4"
server_key_iv: "00215dcb69b56fa3cb7ac9938bd15444"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.330088 27302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "7b815513b7f84c3fb82696f646ec8487"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "4bcf767b026bf1fff177c250a8e53fd4"
server_key_iv: "00215dcb69b56fa3cb7ac9938bd15444"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.331259 27302 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:01.332003 27317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.332202 27302 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:01.332265 27302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "7b815513b7f84c3fb82696f646ec8487"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "4bcf767b026bf1fff177c250a8e53fd4"
server_key_iv: "00215dcb69b56fa3cb7ac9938bd15444"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.332330 27302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:01.350493 27302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:01.352355 27302 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:01.352494 27302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:01.353096 27302 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:01.353140 27302 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.353173 27302 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:01.353188 27302 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:01.360262 27302 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:43801
I20260319 16:33:01.360320 27432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:43801 every 8 connection(s)
I20260319 16:33:01.360608 27302 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:01.361480 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.353467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:52921 (local address 127.24.187.254:35717)
0319 16:33:01.353534 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.353537 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.353772 (+ 235us) server_negotiation.cc:408] Connection header received
0319 16:33:01.354222 (+ 450us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.354224 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.354237 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.354313 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.355112 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.355847 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.356447 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.356644 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.358798 (+ 2154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.358803 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.358803 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.358811 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.360282 (+ 1471us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.360666 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.360667 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.360667 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.360693 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.360894 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.360895 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.360895 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.361007 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.361058 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.361326 (+ 268us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.361415 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:33:01.361513 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.353430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35717 (local address 127.24.187.194:52921)
0319 16:33:01.353595 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.353633 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.354130 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.354405 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.354408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.354748 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:01.355032 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.355036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.355931 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.355932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.356365 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.356368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.356490 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.357099 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:01.357126 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:01.358699 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:01.360379 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.360383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.360385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.360596 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.360748 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.360749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.360750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.360825 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.361058 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:01.361061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:01.361170 (+ 109us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.361267 (+ 97us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.361377 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260319 16:33:01.362233 27433 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35717
I20260319 16:33:01.362349 27433 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:01.362362 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 27302
I20260319 16:33:01.362495 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:01.362586 27433 heartbeater.cc:507] Master 127.24.187.254:35717 requested a full tablet report, sending...
I20260319 16:33:01.362668 25327 external_mini_cluster.cc:1468] Setting key 61e55c512841dbd5db5de87a82cf15fe
I20260319 16:33:01.363125 27086 ts_manager.cc:194] Registered new tserver with Master: 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801)
I20260319 16:33:01.363526 27086 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:52921
I20260319 16:33:01.370287 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.363227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:45483 (local address 127.24.187.254:35717)
0319 16:33:01.363265 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.363267 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.364976 (+ 1709us) server_negotiation.cc:408] Connection header received
0319 16:33:01.365030 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.365032 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.365043 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.365086 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.365658 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.366171 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.366904 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.367022 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.367812 (+ 790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.367816 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.367816 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.367823 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.369105 (+ 1282us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.369440 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.369441 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.369441 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.369463 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.369639 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.369640 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.369641 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.369805 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.369869 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.370137 (+ 268us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.370217 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:33:01.370321 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.363188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35717 (local address 127.24.187.194:45483)
0319 16:33:01.364898 (+ 1710us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.364908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.364974 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.365094 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.365094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.365253 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:01.365553 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.365558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.366250 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.366251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.366814 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.366817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.366896 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.367355 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:01.367360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:01.367718 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:01.369187 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.369188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.369189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.369365 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.369519 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.369519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.369520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.369554 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.369871 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:01.369872 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:01.369940 (+ 68us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.370090 (+ 150us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.370209 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"thread_start_us":1641,"threads_started":1}
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:01.409080 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:35717
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:01.485040 27444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:01.485239 27444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:01.485265 27444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:01.486753 27444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:01.486797 27444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:01.486816 27444 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260319 16:33:01.486827 27444 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:33:01.486850 27444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:01.488446 27444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:01.488631 27444 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:01.488785 27444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:01.491180 27450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:01.491286 27449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:01.491180 27452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:01.491466 27444 server_base.cc:1060] running on GCE node
I20260319 16:33:01.491618 27444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:01.491827 27444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:01.492965 27444 hybrid_clock.cc:648] HybridClock initialized: now 1773937981492949 us; error 30 us; skew 500 ppm
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:01.495622 27444 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:01.496243 27444 webserver.cc:492] Webserver started at http://127.24.187.195:45013/ using document root <none> and password file <none>
I20260319 16:33:01.496447 27444 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:01.496495 27444 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:01.496601 27444 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:01.497809 27444 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "cf1f4292aaeb45d78885afefb33d0a5f"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "02ac4c460b50816c85a8ebe9c388ee40"
server_key_iv: "6c25d39e4371b9b99a37605aac0c3e4d"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.498092 27444 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "cf1f4292aaeb45d78885afefb33d0a5f"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "02ac4c460b50816c85a8ebe9c388ee40"
server_key_iv: "6c25d39e4371b9b99a37605aac0c3e4d"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.499193 27444 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:01.499866 27459 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.500061 27444 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:01.500119 27444 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "cf1f4292aaeb45d78885afefb33d0a5f"
format_stamp: "Formatted at 2026-03-19 16:33:01 on dist-test-slave-svn8"
server_key: "02ac4c460b50816c85a8ebe9c388ee40"
server_key_iv: "6c25d39e4371b9b99a37605aac0c3e4d"
server_key_version: "encryptionkey@0"
I20260319 16:33:01.500185 27444 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:01.532388 27444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:01.534528 27444 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:01.534664 27444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:01.535174 27444 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:01.535211 27444 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.535233 27444 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:01.535244 27444 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:01.541939 27444 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:42855
I20260319 16:33:01.542001 27574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:42855 every 8 connection(s)
I20260319 16:33:01.542301 27444 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:33:01.543704 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.535607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:48545 (local address 127.24.187.254:35717)
0319 16:33:01.535695 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.535698 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.535855 (+ 157us) server_negotiation.cc:408] Connection header received
0319 16:33:01.536398 (+ 543us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.536400 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.536413 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.536470 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.537406 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.538172 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.538915 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.539081 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.541147 (+ 2066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.541152 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.541153 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.541162 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.542628 (+ 1466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.542980 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.542982 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.542982 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.543008 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.543149 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.543149 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.543149 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.543242 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.543286 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.543556 (+ 270us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.543632 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260319 16:33:01.543783 27468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.535582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35717 (local address 127.24.187.195:48545)
0319 16:33:01.535757 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.535786 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.536305 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.536524 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.536528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.536880 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:01.537288 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.537294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.538274 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.538275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.538841 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.538843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.538960 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.539473 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:01.539496 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:01.541051 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:01.542729 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.542732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.542734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.542922 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.543065 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.543065 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.543066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.543113 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.543280 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:01.543282 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:01.543432 (+ 150us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.543518 (+ 86us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.543626 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260319 16:33:01.544219 27575 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35717
I20260319 16:33:01.544298 27575 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:01.544512 27575 heartbeater.cc:507] Master 127.24.187.254:35717 requested a full tablet report, sending...
I20260319 16:33:01.544543 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 27444
I20260319 16:33:01.544606 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:01.544771 25327 external_mini_cluster.cc:1468] Setting key 2886666c217aab46af82c1c3e9a2c46a
I20260319 16:33:01.545120 27085 ts_manager.cc:194] Registered new tserver with Master: cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855)
I20260319 16:33:01.545599 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:33:01.545607 27085 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:48545
I20260319 16:33:01.549911 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.545201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:45903 (local address 127.24.187.254:35717)
0319 16:33:01.545238 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.545240 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.545459 (+ 219us) server_negotiation.cc:408] Connection header received
0319 16:33:01.545507 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.545508 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.545518 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.545569 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.546077 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.546528 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.547032 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.547185 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.547833 (+ 648us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.547836 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.547836 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.547842 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.549006 (+ 1164us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.549277 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.549278 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.549279 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.549299 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.549425 (+ 126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.549426 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.549426 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.549515 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.549570 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.549735 (+ 165us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.549814 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260319 16:33:01.549925 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.545225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35717 (local address 127.24.187.195:45903)
0319 16:33:01.545390 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.545399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.545460 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.545574 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.545575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.545775 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:01.545998 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.546001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.546597 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.546598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.546963 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.546964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.547024 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.547508 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:01.547512 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:01.547753 (+ 241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:01.549081 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.549082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.549083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.549220 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.549345 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.549346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.549346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.549377 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.549566 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:01.549567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:01.549613 (+ 46us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.549671 (+ 58us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.549815 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:01.567301 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.560899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.24.187.254:35717)
0319 16:33:01.560958 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.560959 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.560970 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:01.561009 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.561011 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.561020 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.561060 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.561857 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.562334 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.562978 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.563113 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.565128 (+ 2015us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.565133 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.565133 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.565141 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.566341 (+ 1200us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.566700 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.566702 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.566702 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.566728 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.566860 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.566861 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.566861 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.566948 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.567018 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.567155 (+ 137us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.567230 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260319 16:33:01.568832 27085 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:01.570142 27085 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:01.576212 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.573105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42855 (local address 127.0.0.1:41540)
0319 16:33:01.573498 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.573508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.573555 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.574034 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.574035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.574038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:01.574187 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.574188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.575022 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.575023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.575910 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.575913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.575981 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.576029 (+ 48us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.576077 (+ 48us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.576117 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":78,"threads_started":1}
I20260319 16:33:01.576432 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.572793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.0.0.1:36306)
0319 16:33:01.572987 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.573017 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.573155 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.573477 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.573480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.573484 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:01.573777 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.573780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.575147 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.575147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.575982 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.575984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.576036 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.576038 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.576074 (+ 36us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.576117 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260319 16:33:01.576596 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.572794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43801 (local address 127.0.0.1:33462)
0319 16:33:01.573236 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.573246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.573316 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.573621 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.573623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.573625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:01.573872 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.573875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.575177 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.575177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.575960 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.575963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.576024 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.576034 (+ 10us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.576075 (+ 41us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.576121 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260319 16:33:01.576740 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.573337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.24.187.195:42855)
0319 16:33:01.573674 (+ 337us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.573680 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.573689 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:01.573760 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.573832 (+ 72us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.573851 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.573915 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:01.574255 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.574943 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.576014 (+ 1071us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.576517 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.576574 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.576606 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.576644 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":93,"threads_started":1}
I20260319 16:33:01.577109 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.572911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.24.187.193:33649)
0319 16:33:01.573253 (+ 342us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.573266 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.573276 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:01.573316 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.573318 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.573335 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.573404 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:01.573914 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.575071 (+ 1157us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.576393 (+ 1322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.576919 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.576963 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.576991 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.577039 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":223,"threads_started":1}
I20260319 16:33:01.577298 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.572910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.24.187.194:43801)
0319 16:33:01.573338 (+ 428us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.573347 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.573409 (+ 62us) server_negotiation.cc:408] Connection header received
0319 16:33:01.573451 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.573453 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.573468 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.573526 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:01.573970 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.575080 (+ 1110us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.576181 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.577123 (+ 942us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.577163 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.577192 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.577229 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":101,"threads_started":1}
I20260319 16:33:01.577337 27509 tablet_service.cc:1511] Processing CreateTablet for tablet 0c8288502bb241fbb3f73a2d98bddb13 (DEFAULT_TABLE table=test-table [id=eb4ce76c651f4216abcbe2440e46715a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:01.577626 27509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8288502bb241fbb3f73a2d98bddb13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.577605 27225 tablet_service.cc:1511] Processing CreateTablet for tablet 0c8288502bb241fbb3f73a2d98bddb13 (DEFAULT_TABLE table=test-table [id=eb4ce76c651f4216abcbe2440e46715a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:01.577776 27367 tablet_service.cc:1511] Processing CreateTablet for tablet 0c8288502bb241fbb3f73a2d98bddb13 (DEFAULT_TABLE table=test-table [id=eb4ce76c651f4216abcbe2440e46715a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:01.577875 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8288502bb241fbb3f73a2d98bddb13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.577998 27367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8288502bb241fbb3f73a2d98bddb13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.580399 27598 tablet_bootstrap.cc:492] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Bootstrap starting.
I20260319 16:33:01.580659 27600 tablet_bootstrap.cc:492] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Bootstrap starting.
I20260319 16:33:01.580659 27599 tablet_bootstrap.cc:492] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Bootstrap starting.
I20260319 16:33:01.581039 27598 tablet_bootstrap.cc:654] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.581243 27600 tablet_bootstrap.cc:654] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.581243 27599 tablet_bootstrap.cc:654] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.581517 27599 log.cc:826] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:01.581465 27598 log.cc:826] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:01.581871 27600 log.cc:826] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:01.582298 27599 tablet_bootstrap.cc:492] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: No bootstrap required, opened a new log
I20260319 16:33:01.582371 27599 ts_tablet_manager.cc:1403] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260319 16:33:01.582532 27600 tablet_bootstrap.cc:492] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: No bootstrap required, opened a new log
I20260319 16:33:01.582590 27600 ts_tablet_manager.cc:1403] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:01.582686 27598 tablet_bootstrap.cc:492] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: No bootstrap required, opened a new log
I20260319 16:33:01.582752 27598 ts_tablet_manager.cc:1403] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260319 16:33:01.583810 27600 raft_consensus.cc:359] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.583931 27600 raft_consensus.cc:385] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:01.583985 27600 raft_consensus.cc:740] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b815513b7f84c3fb82696f646ec8487, State: Initialized, Role: FOLLOWER
I20260319 16:33:01.584080 27600 consensus_queue.cc:260] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.584340 27600 ts_tablet_manager.cc:1434] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:01.584254 27598 raft_consensus.cc:359] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.584359 27433 heartbeater.cc:499] Master 127.24.187.254:35717 was elected leader, sending a full tablet report...
I20260319 16:33:01.584383 27598 raft_consensus.cc:385] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:01.584410 27598 raft_consensus.cc:740] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d2e70f3f8d4ab0810c3db8f74e4327, State: Initialized, Role: FOLLOWER
I20260319 16:33:01.584511 27598 consensus_queue.cc:260] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.584254 27599 raft_consensus.cc:359] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.584643 27599 raft_consensus.cc:385] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:01.584692 27599 raft_consensus.cc:740] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1f4292aaeb45d78885afefb33d0a5f, State: Initialized, Role: FOLLOWER
I20260319 16:33:01.584772 27598 ts_tablet_manager.cc:1434] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:01.584808 27599 consensus_queue.cc:260] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.584889 27291 heartbeater.cc:499] Master 127.24.187.254:35717 was elected leader, sending a full tablet report...
I20260319 16:33:01.585044 27599 ts_tablet_manager.cc:1434] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260319 16:33:01.585203 27575 heartbeater.cc:499] Master 127.24.187.254:35717 was elected leader, sending a full tablet report...
W20260319 16:33:01.611212 27434 tablet.cc:2379] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:33:01.680773 27292 tablet.cc:2379] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:33:01.762156 27605 raft_consensus.cc:493] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:01.762287 27605 raft_consensus.cc:515] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.762686 27605 leader_election.cc:290] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855), 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801)
I20260319 16:33:01.765012 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.762883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42855 (local address 127.24.187.193:59879)
0319 16:33:01.763069 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.763081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.763165 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.763293 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.763294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.763296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:01.763492 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.763493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.764134 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.764135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.764747 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.764749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.764822 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.764826 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.764856 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.764896 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":13,"thread_start_us":49,"threads_started":1}
I20260319 16:33:01.765026 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.762883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43801 (local address 127.24.187.193:33781)
0319 16:33:01.763008 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.763021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.763087 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.763214 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.763215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.763217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:01.763395 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.763396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.764115 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.764116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.764730 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.764732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.764796 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.764802 (+ 6us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.764839 (+ 37us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.764878 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260319 16:33:01.765401 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.763034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:59879 (local address 127.24.187.195:42855)
0319 16:33:01.763089 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.763091 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.763147 (+ 56us) server_negotiation.cc:408] Connection header received
0319 16:33:01.763222 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.763223 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.763232 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.763283 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:01.763569 (+ 286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.764049 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.764840 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.765291 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.765301 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.765323 (+ 22us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.765355 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:33:01.765417 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.762952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:33781 (local address 127.24.187.194:43801)
0319 16:33:01.763003 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.763005 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.763084 (+ 79us) server_negotiation.cc:408] Connection header received
0319 16:33:01.763145 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.763146 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.763156 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.763202 (+ 46us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:01.763528 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.764033 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.764822 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.765277 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.765287 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.765313 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.765343 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:33:01.765684 27529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c8288502bb241fbb3f73a2d98bddb13" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" is_pre_election: true
I20260319 16:33:01.765851 27529 raft_consensus.cc:2468] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 0.
I20260319 16:33:01.765817 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c8288502bb241fbb3f73a2d98bddb13" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487" is_pre_election: true
I20260319 16:33:01.765952 27387 raft_consensus.cc:2468] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 0.
I20260319 16:33:01.766090 27176 leader_election.cc:304] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d2e70f3f8d4ab0810c3db8f74e4327, cf1f4292aaeb45d78885afefb33d0a5f; no voters:
I20260319 16:33:01.766274 27605 raft_consensus.cc:2804] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:01.766342 27605 raft_consensus.cc:493] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:01.766359 27605 raft_consensus.cc:3060] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:01.767088 27605 raft_consensus.cc:515] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.767221 27605 leader_election.cc:290] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 election: Requested vote from peers cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855), 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801)
I20260319 16:33:01.767500 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c8288502bb241fbb3f73a2d98bddb13" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487"
I20260319 16:33:01.767521 27529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c8288502bb241fbb3f73a2d98bddb13" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f4292aaeb45d78885afefb33d0a5f"
I20260319 16:33:01.767596 27387 raft_consensus.cc:3060] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:01.767596 27529 raft_consensus.cc:3060] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:01.768472 27387 raft_consensus.cc:2468] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 1.
I20260319 16:33:01.768472 27529 raft_consensus.cc:2468] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 1.
I20260319 16:33:01.768726 27178 leader_election.cc:304] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d2e70f3f8d4ab0810c3db8f74e4327, 7b815513b7f84c3fb82696f646ec8487; no voters:
I20260319 16:33:01.768874 27605 raft_consensus.cc:2804] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:01.768988 27605 raft_consensus.cc:697] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 LEADER]: Becoming Leader. State: Replica: 47d2e70f3f8d4ab0810c3db8f74e4327, State: Running, Role: LEADER
I20260319 16:33:01.769078 27605 consensus_queue.cc:237] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } }
I20260319 16:33:01.769966 27084 catalog_manager.cc:5671] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } health_report { overall_health: UNKNOWN } } }
W20260319 16:33:01.792899 27576 tablet.cc:2379] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Mar 19 16:33:01 dist-test-slave-svn8 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937980, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:01.845893 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.838804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35717 (local address 127.0.0.1:41192)
0319 16:33:01.838979 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.838990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.839074 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.839255 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.839256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.839681 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:01.839899 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.839902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.840641 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.840642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.841022 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.841024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.841089 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.841592 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:01.841596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:01.843209 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:01.844741 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.844748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.844750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.844941 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.845155 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:01.845157 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:01.845157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:01.845190 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:01.845440 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:01.845448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:01.845501 (+ 53us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.845570 (+ 69us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.845802 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260319 16:33:01.846021 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.838892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.24.187.254:35717)
0319 16:33:01.838975 (+ 83us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.838977 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.839070 (+ 93us) server_negotiation.cc:408] Connection header received
0319 16:33:01.839130 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.839132 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.839144 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.839209 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:01.840081 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.840551 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.841121 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.841274 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.843289 (+ 2015us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:01.843295 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:01.843295 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:01.843301 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:01.844636 (+ 1335us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.845046 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.845047 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.845048 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.845071 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:01.845274 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:01.845274 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:01.845275 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:01.845383 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:01.845443 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.845593 (+ 150us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.845880 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":198,"server-negotiator.queue_time_us":58}
I20260319 16:33:01.848412 27084 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:41192:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260319 16:33:01.848637 27084 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:01.850286 27509 tablet_service.cc:1511] Processing CreateTablet for tablet f5c0c7dba45340d48f5894b4fb829758 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=030a8b98527b473190f7e0ca52e9facf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:33:01.850270 27225 tablet_service.cc:1511] Processing CreateTablet for tablet f5c0c7dba45340d48f5894b4fb829758 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=030a8b98527b473190f7e0ca52e9facf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:33:01.850283 27367 tablet_service.cc:1511] Processing CreateTablet for tablet f5c0c7dba45340d48f5894b4fb829758 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=030a8b98527b473190f7e0ca52e9facf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:33:01.850450 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c0c7dba45340d48f5894b4fb829758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.850454 27509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c0c7dba45340d48f5894b4fb829758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.850473 27367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c0c7dba45340d48f5894b4fb829758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.851583 27599 tablet_bootstrap.cc:492] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: Bootstrap starting.
I20260319 16:33:01.851889 27600 tablet_bootstrap.cc:492] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: Bootstrap starting.
I20260319 16:33:01.852013 27599 tablet_bootstrap.cc:654] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.852306 27600 tablet_bootstrap.cc:654] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.852628 27598 tablet_bootstrap.cc:492] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: Bootstrap starting.
I20260319 16:33:01.852839 27599 tablet_bootstrap.cc:492] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: No bootstrap required, opened a new log
I20260319 16:33:01.852881 27599 ts_tablet_manager.cc:1403] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:01.853014 27598 tablet_bootstrap.cc:654] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.853025 27599 raft_consensus.cc:359] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.853104 27599 raft_consensus.cc:385] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:01.853123 27599 raft_consensus.cc:740] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1f4292aaeb45d78885afefb33d0a5f, State: Initialized, Role: FOLLOWER
I20260319 16:33:01.853176 27599 consensus_queue.cc:260] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.853276 27599 ts_tablet_manager.cc:1434] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.853346 27606 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.853438 27606 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:01.853760 27598 tablet_bootstrap.cc:492] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: No bootstrap required, opened a new log
I20260319 16:33:01.853812 27598 ts_tablet_manager.cc:1403] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:01.853956 27598 raft_consensus.cc:359] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.854164 27598 raft_consensus.cc:385] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:01.854187 27598 raft_consensus.cc:740] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d2e70f3f8d4ab0810c3db8f74e4327, State: Initialized, Role: FOLLOWER
I20260319 16:33:01.854252 27600 tablet_bootstrap.cc:492] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: No bootstrap required, opened a new log
I20260319 16:33:01.854241 27598 consensus_queue.cc:260] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.854295 27600 ts_tablet_manager.cc:1403] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:01.854333 27598 ts_tablet_manager.cc:1434] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:01.854395 27605 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.854458 27600 raft_consensus.cc:359] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.854523 27600 raft_consensus.cc:385] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:01.854508 27605 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:01.854542 27600 raft_consensus.cc:740] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b815513b7f84c3fb82696f646ec8487, State: Initialized, Role: FOLLOWER
I20260319 16:33:01.854593 27600 consensus_queue.cc:260] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.854674 27600 ts_tablet_manager.cc:1434] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:01.854725 27604 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.854803 27604 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:01.932778 27605 raft_consensus.cc:493] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:01.932896 27605 raft_consensus.cc:515] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.933068 27605 leader_election.cc:290] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801), cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855)
I20260319 16:33:01.933352 27529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5c0c7dba45340d48f5894b4fb829758" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" is_pre_election: true
I20260319 16:33:01.933348 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5c0c7dba45340d48f5894b4fb829758" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487" is_pre_election: true
I20260319 16:33:01.933467 27529 raft_consensus.cc:2468] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 0.
I20260319 16:33:01.933471 27387 raft_consensus.cc:2468] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 0.
I20260319 16:33:01.933771 27178 leader_election.cc:304] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d2e70f3f8d4ab0810c3db8f74e4327, 7b815513b7f84c3fb82696f646ec8487; no voters:
I20260319 16:33:01.933983 27605 raft_consensus.cc:2804] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:01.934060 27605 raft_consensus.cc:493] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:01.934075 27605 raft_consensus.cc:3060] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:01.934767 27605 raft_consensus.cc:515] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.934913 27605 leader_election.cc:290] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 election: Requested vote from peers 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801), cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855)
I20260319 16:33:01.935091 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5c0c7dba45340d48f5894b4fb829758" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487"
I20260319 16:33:01.935187 27387 raft_consensus.cc:3060] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:01.935298 27529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5c0c7dba45340d48f5894b4fb829758" candidate_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f4292aaeb45d78885afefb33d0a5f"
I20260319 16:33:01.935377 27529 raft_consensus.cc:3060] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:01.936070 27387 raft_consensus.cc:2468] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 1.
I20260319 16:33:01.936303 27178 leader_election.cc:304] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d2e70f3f8d4ab0810c3db8f74e4327, 7b815513b7f84c3fb82696f646ec8487; no voters:
I20260319 16:33:01.936451 27529 raft_consensus.cc:2468] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d2e70f3f8d4ab0810c3db8f74e4327 in term 1.
I20260319 16:33:01.936504 27605 raft_consensus.cc:2804] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:01.936563 27605 raft_consensus.cc:697] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 LEADER]: Becoming Leader. State: Replica: 47d2e70f3f8d4ab0810c3db8f74e4327, State: Running, Role: LEADER
I20260319 16:33:01.936708 27605 consensus_queue.cc:237] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.937330 27609 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: TxnStatusTablet state changed. Reason: New leader 47d2e70f3f8d4ab0810c3db8f74e4327. Latest consensus state: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.937515 27609 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:33:01.937875 27621 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260319 16:33:01.937978 27621 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260319 16:33:01.937937 27085 catalog_manager.cc:5671] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } health_report { overall_health: UNKNOWN } } }
I20260319 16:33:01.950423 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.948147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.0.0.1:36316)
0319 16:33:01.948249 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.948259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.948299 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.948498 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.948499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.948502 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:01.948662 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.948664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.949279 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.949280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.949737 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.949740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.949802 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.949825 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:01.950233 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:01.950244 (+ 11us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.950295 (+ 51us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.950349 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260319 16:33:01.950407 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.948287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.24.187.193:33649)
0319 16:33:01.948346 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.948348 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.948355 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:33:01.948389 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.948390 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.948406 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.948459 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:01.948727 (+ 268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.949209 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.949805 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.950037 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.950072 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:01.950160 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:01.950248 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.950300 (+ 52us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.950325 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260319 16:33:01.952013 27387 raft_consensus.cc:1275] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Refusing update from remote peer 47d2e70f3f8d4ab0810c3db8f74e4327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:01.952040 27529 raft_consensus.cc:1275] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Refusing update from remote peer 47d2e70f3f8d4ab0810c3db8f74e4327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:01.952301 27609 consensus_queue.cc:1048] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:01.952464 27609 consensus_queue.cc:1048] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:01.953403 27604 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: TxnStatusTablet state changed. Reason: New leader 47d2e70f3f8d4ab0810c3db8f74e4327. Latest consensus state: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.953481 27604 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:01.954493 27606 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: TxnStatusTablet state changed. Reason: New leader 47d2e70f3f8d4ab0810c3db8f74e4327. Latest consensus state: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.954530 27605 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.954588 27606 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:01.954592 27605 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:33:01.954669 27624 mvcc.cc:204] Tried to move back new op lower bound from 7266049974073131008 to 7266049974013759488. Current Snapshot: MvccSnapshot[applied={T|T < 7266049974073131008}]
I20260319 16:33:01.955159 27604 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.955240 27604 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P 7b815513b7f84c3fb82696f646ec8487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:01.955603 27609 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.955687 27609 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P 47d2e70f3f8d4ab0810c3db8f74e4327: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:33:01.956048 27625 mvcc.cc:204] Tried to move back new op lower bound from 7266049974073131008 to 7266049974013759488. Current Snapshot: MvccSnapshot[applied={T|T < 7266049974073131008 or (T in {7266049974073131008})}]
I20260319 16:33:01.956471 27606 tablet_replica.cc:442] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } } }
I20260319 16:33:01.956557 27606 tablet_replica.cc:445] T f5c0c7dba45340d48f5894b4fb829758 P cf1f4292aaeb45d78885afefb33d0a5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:01.961086 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.958692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.24.187.193:33649)
0319 16:33:01.958738 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.958740 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.958748 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:01.958807 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.958808 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.958817 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.958862 (+ 45us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:01.959159 (+ 297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.959614 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.960479 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.960608 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.960634 (+ 26us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:01.960660 (+ 26us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:01.960705 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.960957 (+ 252us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.960983 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:33:01.966836 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.964168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:45849 (local address 127.24.187.193:33649)
0319 16:33:01.964229 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:01.964231 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:01.964414 (+ 183us) server_negotiation.cc:408] Connection header received
0319 16:33:01.964437 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:01.964438 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:01.964447 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:01.964494 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:01.964876 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.965387 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.966290 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.966508 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.966535 (+ 27us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:01.966561 (+ 26us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:01.966611 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:01.966674 (+ 63us) server_negotiation.cc:300] Negotiation successful
0319 16:33:01.966746 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:33:01.966861 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:01.964108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.24.187.193:45849)
0319 16:33:01.964303 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:01.964314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:01.964384 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:01.964507 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:01.964508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:01.964510 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:01.964770 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.964772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.965467 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:01.965468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:01.966182 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:01.966184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:01.966258 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:01.966288 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:01.966626 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:01.966628 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:01.966665 (+ 37us) client_negotiation.cc:240] Negotiation successful
0319 16:33:01.966709 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260319 16:33:01.969432 27387 raft_consensus.cc:1275] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Refusing update from remote peer 47d2e70f3f8d4ab0810c3db8f74e4327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:01.969496 27529 raft_consensus.cc:1275] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Refusing update from remote peer 47d2e70f3f8d4ab0810c3db8f74e4327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:01.969696 27605 consensus_queue.cc:1048] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:01.969897 27609 consensus_queue.cc:1048] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:01.973265 27627 mvcc.cc:204] Tried to move back new op lower bound from 7266049974145142784 to 7266049973326643200. Current Snapshot: MvccSnapshot[applied={T|T < 7266049974145142784}]
I20260319 16:33:01.989967 27084 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:01.990218 27084 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:01.991357 27084 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Sending DeleteTablet for 3 replicas of tablet 0c8288502bb241fbb3f73a2d98bddb13
I20260319 16:33:01.991811 27367 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c8288502bb241fbb3f73a2d98bddb13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:01 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33462
I20260319 16:33:01.991950 27509 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c8288502bb241fbb3f73a2d98bddb13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:01 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260319 16:33:01.992277 27643 tablet_replica.cc:333] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: stopping tablet replica
I20260319 16:33:01.992367 27225 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c8288502bb241fbb3f73a2d98bddb13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:01 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260319 16:33:01.992434 27643 raft_consensus.cc:2243] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:01.992612 27643 raft_consensus.cc:2272] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:01.992636 27084 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:01.992817 27084 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:01.993111 27643 ts_tablet_manager.cc:1916] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:01.993161 27644 tablet_replica.cc:333] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: stopping tablet replica
I20260319 16:33:01.993261 27644 raft_consensus.cc:2243] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:01.993371 27644 raft_consensus.cc:2272] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:01.993916 27644 ts_tablet_manager.cc:1916] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:01.994843 27367 tablet_service.cc:1511] Processing CreateTablet for tablet ffe0fac1c91d4a49915d84a1cac72a24 (DEFAULT_TABLE table=test-table [id=c330c46497904fd5abc3f3883afdb3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:01.994885 27225 tablet_service.cc:1511] Processing CreateTablet for tablet ffe0fac1c91d4a49915d84a1cac72a24 (DEFAULT_TABLE table=test-table [id=c330c46497904fd5abc3f3883afdb3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:01.994974 27367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe0fac1c91d4a49915d84a1cac72a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.995043 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe0fac1c91d4a49915d84a1cac72a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.995270 27643 ts_tablet_manager.cc:1929] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:01.995389 27643 log.cc:1199] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/0c8288502bb241fbb3f73a2d98bddb13
I20260319 16:33:01.995682 27643 ts_tablet_manager.cc:1950] T 0c8288502bb241fbb3f73a2d98bddb13 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting consensus metadata
I20260319 16:33:01.996101 27070 catalog_manager.cc:5002] TS cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855): tablet 0c8288502bb241fbb3f73a2d98bddb13 (table test-table [id=eb4ce76c651f4216abcbe2440e46715a]) successfully deleted
I20260319 16:33:01.996249 27600 tablet_bootstrap.cc:492] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: Bootstrap starting.
I20260319 16:33:01.996757 27509 tablet_service.cc:1511] Processing CreateTablet for tablet ffe0fac1c91d4a49915d84a1cac72a24 (DEFAULT_TABLE table=test-table [id=c330c46497904fd5abc3f3883afdb3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:01.996930 27509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe0fac1c91d4a49915d84a1cac72a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:01.996973 27600 tablet_bootstrap.cc:654] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.998292 27600 tablet_bootstrap.cc:492] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: No bootstrap required, opened a new log
I20260319 16:33:01.998371 27600 ts_tablet_manager.cc:1403] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:01.998421 27598 tablet_bootstrap.cc:492] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: Bootstrap starting.
I20260319 16:33:01.998632 27600 raft_consensus.cc:359] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.998724 27600 raft_consensus.cc:385] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:01.998744 27600 raft_consensus.cc:740] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b815513b7f84c3fb82696f646ec8487, State: Initialized, Role: FOLLOWER
I20260319 16:33:01.998791 27600 consensus_queue.cc:260] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:01.998879 27600 ts_tablet_manager.cc:1434] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:01.998950 27598 tablet_bootstrap.cc:654] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:01.999279 27599 tablet_bootstrap.cc:492] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: Bootstrap starting.
I20260319 16:33:01.999691 27599 tablet_bootstrap.cc:654] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:02.000319 27599 tablet_bootstrap.cc:492] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: No bootstrap required, opened a new log
I20260319 16:33:02.000371 27599 ts_tablet_manager.cc:1403] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:02.000523 27599 raft_consensus.cc:359] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:02.000591 27599 raft_consensus.cc:385] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:02.000610 27599 raft_consensus.cc:740] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1f4292aaeb45d78885afefb33d0a5f, State: Initialized, Role: FOLLOWER
I20260319 16:33:02.000659 27599 consensus_queue.cc:260] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:02.000751 27599 ts_tablet_manager.cc:1434] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:02.001425 27598 tablet_bootstrap.cc:492] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: No bootstrap required, opened a new log
I20260319 16:33:02.001480 27598 ts_tablet_manager.cc:1403] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260319 16:33:02.001633 27598 raft_consensus.cc:359] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:02.001703 27598 raft_consensus.cc:385] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:02.001754 27598 raft_consensus.cc:740] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d2e70f3f8d4ab0810c3db8f74e4327, State: Initialized, Role: FOLLOWER
I20260319 16:33:02.001854 27598 consensus_queue.cc:260] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:02.001953 27598 ts_tablet_manager.cc:1434] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:02.001742 27644 ts_tablet_manager.cc:1929] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:02.002120 27644 log.cc:1199] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/0c8288502bb241fbb3f73a2d98bddb13
I20260319 16:33:02.002367 27644 ts_tablet_manager.cc:1950] T 0c8288502bb241fbb3f73a2d98bddb13 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting consensus metadata
I20260319 16:33:02.002746 27070 catalog_manager.cc:5002] TS 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649): tablet 0c8288502bb241fbb3f73a2d98bddb13 (table test-table [id=eb4ce76c651f4216abcbe2440e46715a]) successfully deleted
I20260319 16:33:02.005862 27642 tablet_replica.cc:333] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: stopping tablet replica
I20260319 16:33:02.005996 27642 raft_consensus.cc:2243] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:02.006079 27642 raft_consensus.cc:2272] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:02.006752 27642 ts_tablet_manager.cc:1916] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:02.009037 27642 ts_tablet_manager.cc:1929] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:02.009183 27642 log.cc:1199] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/0c8288502bb241fbb3f73a2d98bddb13
I20260319 16:33:02.009593 27642 ts_tablet_manager.cc:1950] T 0c8288502bb241fbb3f73a2d98bddb13 P 7b815513b7f84c3fb82696f646ec8487: Deleting consensus metadata
I20260319 16:33:02.010617 27072 catalog_manager.cc:5002] TS 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801): tablet 0c8288502bb241fbb3f73a2d98bddb13 (table test-table [id=eb4ce76c651f4216abcbe2440e46715a]) successfully deleted
I20260319 16:33:02.036269 27604 raft_consensus.cc:493] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:02.036494 27604 raft_consensus.cc:515] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:02.036932 27604 leader_election.cc:290] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649), cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855)
I20260319 16:33:02.039820 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.037372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42855 (local address 127.24.187.194:45765)
0319 16:33:02.037494 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:02.037505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:02.037568 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:02.037737 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:02.037738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:02.037741 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:02.037999 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.038000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.038678 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.038679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:02.039573 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.039576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.039650 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.039655 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:02.039686 (+ 31us) client_negotiation.cc:240] Negotiation successful
0319 16:33:02.039725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":34,"threads_started":1}
I20260319 16:33:02.040279 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.037520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:45765 (local address 127.24.187.195:42855)
0319 16:33:02.037557 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:02.037559 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:02.037569 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:02.037623 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:02.037625 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:02.037635 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:02.037694 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:02.038080 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.038597 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.039686 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.040153 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.040162 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:02.040187 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:33:02.040220 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:33:02.040441 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.037194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.24.187.194:45133)
0319 16:33:02.037466 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:02.037480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:02.037614 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:02.037818 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:02.037819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:02.037824 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:02.038018 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.038019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.038925 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.038925 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:02.039712 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.039714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.040279 (+ 565us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.040283 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:02.040322 (+ 39us) client_negotiation.cc:240] Negotiation successful
0319 16:33:02.040369 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":73,"threads_started":1}
I20260319 16:33:02.040520 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.037310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:45133 (local address 127.24.187.193:33649)
0319 16:33:02.037364 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:02.037366 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:02.037579 (+ 213us) server_negotiation.cc:408] Connection header received
0319 16:33:02.037754 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:02.037755 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:02.037765 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:02.037811 (+ 46us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:02.038120 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.038845 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.039801 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.040173 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.040180 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:02.040409 (+ 229us) server_negotiation.cc:300] Negotiation successful
0319 16:33:02.040464 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:33:02.040963 27245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe0fac1c91d4a49915d84a1cac72a24" candidate_uuid: "7b815513b7f84c3fb82696f646ec8487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" is_pre_election: true
I20260319 16:33:02.041090 27245 raft_consensus.cc:2468] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b815513b7f84c3fb82696f646ec8487 in term 0.
I20260319 16:33:02.041289 27318 leader_election.cc:304] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d2e70f3f8d4ab0810c3db8f74e4327, 7b815513b7f84c3fb82696f646ec8487; no voters:
I20260319 16:33:02.044396 27604 raft_consensus.cc:2804] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:02.044468 27604 raft_consensus.cc:493] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:02.044485 27604 raft_consensus.cc:3060] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:02.045315 27604 raft_consensus.cc:515] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:02.045475 27604 leader_election.cc:290] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [CANDIDATE]: Term 1 election: Requested vote from peers 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649), cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855)
I20260319 16:33:02.045581 27529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe0fac1c91d4a49915d84a1cac72a24" candidate_uuid: "7b815513b7f84c3fb82696f646ec8487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" is_pre_election: true
I20260319 16:33:02.045668 27529 raft_consensus.cc:2468] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b815513b7f84c3fb82696f646ec8487 in term 0.
I20260319 16:33:02.046366 27529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe0fac1c91d4a49915d84a1cac72a24" candidate_uuid: "7b815513b7f84c3fb82696f646ec8487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf1f4292aaeb45d78885afefb33d0a5f"
I20260319 16:33:02.046598 27529 raft_consensus.cc:3060] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:02.047220 27529 raft_consensus.cc:2468] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b815513b7f84c3fb82696f646ec8487 in term 1.
I20260319 16:33:02.047472 27318 leader_election.cc:304] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b815513b7f84c3fb82696f646ec8487, cf1f4292aaeb45d78885afefb33d0a5f; no voters:
I20260319 16:33:02.047461 27245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffe0fac1c91d4a49915d84a1cac72a24" candidate_uuid: "7b815513b7f84c3fb82696f646ec8487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327"
I20260319 16:33:02.047547 27245 raft_consensus.cc:3060] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:02.047590 27604 raft_consensus.cc:2804] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:02.047883 27604 raft_consensus.cc:697] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 1 LEADER]: Becoming Leader. State: Replica: 7b815513b7f84c3fb82696f646ec8487, State: Running, Role: LEADER
I20260319 16:33:02.047973 27604 consensus_queue.cc:237] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:02.048323 27245 raft_consensus.cc:2468] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b815513b7f84c3fb82696f646ec8487 in term 1.
I20260319 16:33:02.048739 27084 catalog_manager.cc:5671] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "7b815513b7f84c3fb82696f646ec8487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } health_report { overall_health: UNKNOWN } } }
I20260319 16:33:02.067059 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.063338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.24.187.194:43801)
0319 16:33:02.063415 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:02.063417 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:02.063428 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:02.063475 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:02.063476 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:02.063501 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:02.063712 (+ 211us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:02.064074 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.065030 (+ 956us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.066033 (+ 1003us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.066495 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.066563 (+ 68us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:02.066728 (+ 165us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:02.066830 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:02.066936 (+ 106us) server_negotiation.cc:300] Negotiation successful
0319 16:33:02.066973 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260319 16:33:02.072656 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.070055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:44689 (local address 127.24.187.193:33649)
0319 16:33:02.070100 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:02.070102 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:02.070202 (+ 100us) server_negotiation.cc:408] Connection header received
0319 16:33:02.070247 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:02.070248 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:02.070258 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:02.070305 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:02.070730 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.071294 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.072160 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.072312 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.072330 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:02.072357 (+ 27us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:02.072417 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:02.072519 (+ 102us) server_negotiation.cc:300] Negotiation successful
0319 16:33:02.072546 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:33:02.072665 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.069957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.24.187.194:44689)
0319 16:33:02.070109 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:02.070121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:02.070193 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:02.070311 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:02.070312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:02.070314 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:02.070580 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.070583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.071400 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.071401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:02.072054 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.072057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.072138 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.072162 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:02.072449 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:02.072452 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:02.072510 (+ 58us) client_negotiation.cc:240] Negotiation successful
0319 16:33:02.072559 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260319 16:33:02.075426 27529 raft_consensus.cc:1275] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Refusing update from remote peer 7b815513b7f84c3fb82696f646ec8487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:02.075532 27245 raft_consensus.cc:1275] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Refusing update from remote peer 7b815513b7f84c3fb82696f646ec8487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:02.075713 27604 consensus_queue.cc:1048] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:02.075867 27650 consensus_queue.cc:1048] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:02.084131 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.081669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43801 (local address 127.24.187.193:55447)
0319 16:33:02.081755 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:02.081768 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:02.081815 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:02.081951 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:02.081952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:02.081953 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:02.082253 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.082257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.082922 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.082924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:02.083544 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:02.083547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.083645 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.083652 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:02.083964 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:02.083966 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:02.084008 (+ 42us) client_negotiation.cc:240] Negotiation successful
0319 16:33:02.084056 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260319 16:33:02.084233 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:02.081812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:55447 (local address 127.24.187.194:43801)
0319 16:33:02.081852 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:02.081854 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:02.081862 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:02.081882 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:02.081883 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:02.081891 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:02.081934 (+ 43us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:02.082345 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.082839 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:02.083649 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:02.083835 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:02.083859 (+ 24us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:02.083885 (+ 26us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:02.083969 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:02.084141 (+ 172us) server_negotiation.cc:300] Negotiation successful
0319 16:33:02.084170 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260319 16:33:02.084578 27625 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260319 16:33:03.605125 27085 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:03.605229 27085 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:03.606098 27085 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Sending DeleteTablet for 3 replicas of tablet ffe0fac1c91d4a49915d84a1cac72a24
I20260319 16:33:03.606338 27367 tablet_service.cc:1558] Processing DeleteTablet for tablet ffe0fac1c91d4a49915d84a1cac72a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:03 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33462
I20260319 16:33:03.606377 27509 tablet_service.cc:1558] Processing DeleteTablet for tablet ffe0fac1c91d4a49915d84a1cac72a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:03 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260319 16:33:03.606524 27225 tablet_service.cc:1558] Processing DeleteTablet for tablet ffe0fac1c91d4a49915d84a1cac72a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:03 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260319 16:33:03.608911 27681 tablet_replica.cc:333] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: stopping tablet replica
I20260319 16:33:03.608971 27681 raft_consensus.cc:2243] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:03.609001 27681 raft_consensus.cc:2272] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:03.609103 27680 tablet_replica.cc:333] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: stopping tablet replica
I20260319 16:33:03.609229 27682 tablet_replica.cc:333] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: stopping tablet replica
I20260319 16:33:03.609246 27681 ts_tablet_manager.cc:1916] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:03.609287 27682 raft_consensus.cc:2243] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:03.609364 27680 raft_consensus.cc:2243] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:03.609470 27680 raft_consensus.cc:2272] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:03.609551 27682 raft_consensus.cc:2272] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:03.609836 27682 ts_tablet_manager.cc:1916] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:03.609845 27680 ts_tablet_manager.cc:1916] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:03.611112 27681 ts_tablet_manager.cc:1929] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260319 16:33:03.611232 27681 log.cc:1199] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/ffe0fac1c91d4a49915d84a1cac72a24
I20260319 16:33:03.611366 27680 ts_tablet_manager.cc:1929] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260319 16:33:03.611456 27680 log.cc:1199] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/ffe0fac1c91d4a49915d84a1cac72a24
I20260319 16:33:03.611526 27681 ts_tablet_manager.cc:1950] T ffe0fac1c91d4a49915d84a1cac72a24 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting consensus metadata
I20260319 16:33:03.611591 27682 ts_tablet_manager.cc:1929] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260319 16:33:03.611683 27682 log.cc:1199] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/ffe0fac1c91d4a49915d84a1cac72a24
I20260319 16:33:03.611791 27680 ts_tablet_manager.cc:1950] T ffe0fac1c91d4a49915d84a1cac72a24 P 7b815513b7f84c3fb82696f646ec8487: Deleting consensus metadata
I20260319 16:33:03.611884 27070 catalog_manager.cc:5002] TS cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855): tablet ffe0fac1c91d4a49915d84a1cac72a24 (table test-table [id=c330c46497904fd5abc3f3883afdb3f2]) successfully deleted
I20260319 16:33:03.611943 27682 ts_tablet_manager.cc:1950] T ffe0fac1c91d4a49915d84a1cac72a24 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting consensus metadata
I20260319 16:33:03.612232 27072 catalog_manager.cc:5002] TS 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801): tablet ffe0fac1c91d4a49915d84a1cac72a24 (table test-table [id=c330c46497904fd5abc3f3883afdb3f2]) successfully deleted
I20260319 16:33:03.612239 27070 catalog_manager.cc:5002] TS 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649): tablet ffe0fac1c91d4a49915d84a1cac72a24 (table test-table [id=c330c46497904fd5abc3f3883afdb3f2]) successfully deleted
Mar 19 16:33:03 dist-test-slave-svn8 krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 19 16:33:03 dist-test-slave-svn8 krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:03.628937 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.622729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.24.187.254:35717)
0319 16:33:03.622864 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:03.622866 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:03.622879 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:33:03.622960 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:03.622961 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:03.622974 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:03.623027 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:03.623683 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.624196 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.624760 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.624922 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.626906 (+ 1984us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:03.626910 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:03.626910 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:03.626917 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:03.628063 (+ 1146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:03.628323 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:03.628324 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:03.628324 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:03.628344 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:03.628475 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:03.628476 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:03.628476 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:03.628563 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:03.628653 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:03.628747 (+ 94us) server_negotiation.cc:300] Negotiation successful
0319 16:33:03.628836 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260319 16:33:03.630389 27085 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:03.630569 27085 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:03.631901 27225 tablet_service.cc:1511] Processing CreateTablet for tablet c16410dc2f9f42198defe229dcf89234 (DEFAULT_TABLE table=test-table [id=575f9c341b394fbcb6c1409de3d28dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:03.631913 27367 tablet_service.cc:1511] Processing CreateTablet for tablet c16410dc2f9f42198defe229dcf89234 (DEFAULT_TABLE table=test-table [id=575f9c341b394fbcb6c1409de3d28dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:03.632022 27367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c16410dc2f9f42198defe229dcf89234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:03.632033 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c16410dc2f9f42198defe229dcf89234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:03.633023 27691 tablet_bootstrap.cc:492] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: Bootstrap starting.
I20260319 16:33:03.633049 27692 tablet_bootstrap.cc:492] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: Bootstrap starting.
I20260319 16:33:03.633095 27509 tablet_service.cc:1511] Processing CreateTablet for tablet c16410dc2f9f42198defe229dcf89234 (DEFAULT_TABLE table=test-table [id=575f9c341b394fbcb6c1409de3d28dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:03.633221 27509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c16410dc2f9f42198defe229dcf89234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:03.633476 27692 tablet_bootstrap.cc:654] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:03.633513 27691 tablet_bootstrap.cc:654] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:03.634115 27691 tablet_bootstrap.cc:492] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: No bootstrap required, opened a new log
I20260319 16:33:03.634159 27691 ts_tablet_manager.cc:1403] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:03.634270 27691 raft_consensus.cc:359] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.634301 27692 tablet_bootstrap.cc:492] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: No bootstrap required, opened a new log
I20260319 16:33:03.634320 27691 raft_consensus.cc:385] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:03.634338 27691 raft_consensus.cc:740] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b815513b7f84c3fb82696f646ec8487, State: Initialized, Role: FOLLOWER
I20260319 16:33:03.634346 27692 ts_tablet_manager.cc:1403] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:03.634387 27695 tablet_bootstrap.cc:492] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: Bootstrap starting.
I20260319 16:33:03.634418 27691 consensus_queue.cc:260] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.634452 27692 raft_consensus.cc:359] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.634514 27692 raft_consensus.cc:385] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:03.634533 27692 raft_consensus.cc:740] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d2e70f3f8d4ab0810c3db8f74e4327, State: Initialized, Role: FOLLOWER
I20260319 16:33:03.634610 27692 consensus_queue.cc:260] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.634719 27692 ts_tablet_manager.cc:1434] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:03.634758 27695 tablet_bootstrap.cc:654] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:03.634851 27691 ts_tablet_manager.cc:1434] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260319 16:33:03.635432 27695 tablet_bootstrap.cc:492] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: No bootstrap required, opened a new log
I20260319 16:33:03.635509 27695 ts_tablet_manager.cc:1403] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:03.635617 27695 raft_consensus.cc:359] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.635669 27695 raft_consensus.cc:385] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:03.635682 27695 raft_consensus.cc:740] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1f4292aaeb45d78885afefb33d0a5f, State: Initialized, Role: FOLLOWER
I20260319 16:33:03.635716 27695 consensus_queue.cc:260] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.635870 27695 ts_tablet_manager.cc:1434] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:03.815218 27697 raft_consensus.cc:493] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:03.815323 27697 raft_consensus.cc:515] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.815667 27697 leader_election.cc:290] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649), 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801)
I20260319 16:33:03.818157 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.815864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43801 (local address 127.24.187.195:43673)
0319 16:33:03.816097 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:03.816110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:03.816186 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:03.816330 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:03.816331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:03.816334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:03.816561 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.816563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.817268 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.817269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:03.817964 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.817967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.818023 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.818028 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:03.818048 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:33:03.818078 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":167,"threads_started":1}
I20260319 16:33:03.818288 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.815864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.24.187.195:52663)
0319 16:33:03.816173 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:03.816184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:03.816256 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:03.816465 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:03.816466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:03.816468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:03.816717 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.816719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.817433 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.817434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:03.818064 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.818066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.818141 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.818144 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:03.818174 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:33:03.818211 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":11,"thread_start_us":154,"threads_started":1}
I20260319 16:33:03.818614 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.815934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:43673 (local address 127.24.187.194:43801)
0319 16:33:03.816103 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:03.816105 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:03.816188 (+ 83us) server_negotiation.cc:408] Connection header received
0319 16:33:03.816240 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:03.816241 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:03.816253 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:03.816316 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:03.816641 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.817192 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.818067 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.818516 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.818524 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:03.818544 (+ 20us) server_negotiation.cc:300] Negotiation successful
0319 16:33:03.818566 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260319 16:33:03.818840 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.815986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:52663 (local address 127.24.187.193:33649)
0319 16:33:03.816165 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:03.816167 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:03.816260 (+ 93us) server_negotiation.cc:408] Connection header received
0319 16:33:03.816325 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:03.816327 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:03.816339 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:03.816459 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:03.816794 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.817311 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.818155 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.818694 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.818704 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:03.818732 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:33:03.818762 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260319 16:33:03.818821 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c16410dc2f9f42198defe229dcf89234" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487" is_pre_election: true
I20260319 16:33:03.818900 27387 raft_consensus.cc:2468] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 0.
I20260319 16:33:03.819059 27245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c16410dc2f9f42198defe229dcf89234" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" is_pre_election: true
I20260319 16:33:03.819151 27245 raft_consensus.cc:2468] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 0.
I20260319 16:33:03.819131 27462 leader_election.cc:304] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b815513b7f84c3fb82696f646ec8487, cf1f4292aaeb45d78885afefb33d0a5f; no voters:
I20260319 16:33:03.819276 27697 raft_consensus.cc:2804] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:03.819315 27697 raft_consensus.cc:493] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:03.819334 27697 raft_consensus.cc:3060] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:03.820026 27697 raft_consensus.cc:515] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.820150 27697 leader_election.cc:290] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 election: Requested vote from peers 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649), 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801)
I20260319 16:33:03.820291 27245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c16410dc2f9f42198defe229dcf89234" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327"
I20260319 16:33:03.820360 27245 raft_consensus.cc:3060] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:03.820356 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c16410dc2f9f42198defe229dcf89234" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487"
I20260319 16:33:03.820441 27387 raft_consensus.cc:3060] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:03.820945 27245 raft_consensus.cc:2468] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 1.
I20260319 16:33:03.821082 27387 raft_consensus.cc:2468] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 1.
I20260319 16:33:03.821430 27460 leader_election.cc:304] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d2e70f3f8d4ab0810c3db8f74e4327, cf1f4292aaeb45d78885afefb33d0a5f; no voters:
I20260319 16:33:03.821563 27697 raft_consensus.cc:2804] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:03.821749 27697 raft_consensus.cc:697] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 LEADER]: Becoming Leader. State: Replica: cf1f4292aaeb45d78885afefb33d0a5f, State: Running, Role: LEADER
I20260319 16:33:03.821823 27697 consensus_queue.cc:237] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.822468 27085 catalog_manager.cc:5671] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f reported cstate change: term changed from 0 to 1, leader changed from <none> to cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195). New cstate: current_term: 1 leader_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } health_report { overall_health: HEALTHY } } }
I20260319 16:33:03.865474 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.863151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.24.187.195:42855)
0319 16:33:03.863315 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:03.863318 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:03.863329 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:03.863374 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:03.863375 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:03.863392 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:03.863463 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:03.863754 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.864244 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.864918 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.865063 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.865097 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:03.865189 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:03.865255 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:03.865367 (+ 112us) server_negotiation.cc:300] Negotiation successful
0319 16:33:03.865391 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260319 16:33:03.869848 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.867893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:56635 (local address 127.24.187.193:33649)
0319 16:33:03.867946 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:03.867948 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:03.868046 (+ 98us) server_negotiation.cc:408] Connection header received
0319 16:33:03.868108 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:03.868109 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:03.868119 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:03.868162 (+ 43us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:03.868410 (+ 248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.868853 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.869403 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.869557 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.869576 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:03.869598 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:03.869641 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:03.869751 (+ 110us) server_negotiation.cc:300] Negotiation successful
0319 16:33:03.869776 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:03.869863 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.867767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.24.187.195:56635)
0319 16:33:03.867985 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:03.867994 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:03.868052 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:03.868174 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:03.868175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:03.868176 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:03.868346 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.868347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.868920 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.868921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:03.869335 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.869337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.869396 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.869451 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:03.869655 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:03.869658 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:03.869740 (+ 82us) client_negotiation.cc:240] Negotiation successful
0319 16:33:03.869784 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
I20260319 16:33:03.871676 27387 raft_consensus.cc:1275] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Refusing update from remote peer cf1f4292aaeb45d78885afefb33d0a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:03.871680 27245 raft_consensus.cc:1275] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Refusing update from remote peer cf1f4292aaeb45d78885afefb33d0a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:03.871893 27702 consensus_queue.cc:1048] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:03.871999 27697 consensus_queue.cc:1048] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:03.878635 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.876690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:59921 (local address 127.24.187.195:42855)
0319 16:33:03.876727 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:03.876729 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:03.876821 (+ 92us) server_negotiation.cc:408] Connection header received
0319 16:33:03.876880 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:03.876881 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:03.876893 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:03.876938 (+ 45us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:03.877161 (+ 223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.877641 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.878200 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.878332 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.878348 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:03.878382 (+ 34us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:03.878447 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:03.878529 (+ 82us) server_negotiation.cc:300] Negotiation successful
0319 16:33:03.878552 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:33:03.878635 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:03.876581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42855 (local address 127.24.187.193:59921)
0319 16:33:03.876757 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:03.876767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:03.876824 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:03.876939 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:03.876940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:03.876942 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:03.877113 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.877114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.877739 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:03.877739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:03.878111 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:03.878112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:03.878167 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:03.878172 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:03.878467 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:03.878470 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:03.878518 (+ 48us) client_negotiation.cc:240] Negotiation successful
0319 16:33:03.878569 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260319 16:33:03.885346 27085 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:03.885449 27085 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:03.886159 27085 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Sending DeleteTablet for 3 replicas of tablet c16410dc2f9f42198defe229dcf89234
I20260319 16:33:03.886401 27225 tablet_service.cc:1558] Processing DeleteTablet for tablet c16410dc2f9f42198defe229dcf89234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:03 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260319 16:33:03.886492 27682 tablet_replica.cc:333] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: stopping tablet replica
I20260319 16:33:03.886479 27509 tablet_service.cc:1558] Processing DeleteTablet for tablet c16410dc2f9f42198defe229dcf89234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:03 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260319 16:33:03.886485 27367 tablet_service.cc:1558] Processing DeleteTablet for tablet c16410dc2f9f42198defe229dcf89234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:03 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33462
I20260319 16:33:03.886539 27682 raft_consensus.cc:2243] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:03.886567 27680 tablet_replica.cc:333] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: stopping tablet replica
I20260319 16:33:03.886572 27682 raft_consensus.cc:2272] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:03.886572 27681 tablet_replica.cc:333] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: stopping tablet replica
I20260319 16:33:03.886608 27680 raft_consensus.cc:2243] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:03.886621 27681 raft_consensus.cc:2243] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:03.886641 27680 raft_consensus.cc:2272] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:03.886694 27681 raft_consensus.cc:2272] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:03.886977 27681 ts_tablet_manager.cc:1916] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:03.887024 27680 ts_tablet_manager.cc:1916] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:03.887054 27682 ts_tablet_manager.cc:1916] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:03.888876 27680 ts_tablet_manager.cc:1929] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:03.888938 27680 log.cc:1199] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/c16410dc2f9f42198defe229dcf89234
I20260319 16:33:03.888911 27085 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260319 16:33:03.889004 27682 ts_tablet_manager.cc:1929] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:03.889057 27682 log.cc:1199] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/c16410dc2f9f42198defe229dcf89234
W20260319 16:33:03.889062 27085 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:03.889138 27680 ts_tablet_manager.cc:1950] T c16410dc2f9f42198defe229dcf89234 P 7b815513b7f84c3fb82696f646ec8487: Deleting consensus metadata
I20260319 16:33:03.889287 27682 ts_tablet_manager.cc:1950] T c16410dc2f9f42198defe229dcf89234 P 47d2e70f3f8d4ab0810c3db8f74e4327: Deleting consensus metadata
I20260319 16:33:03.889447 27072 catalog_manager.cc:5002] TS 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801): tablet c16410dc2f9f42198defe229dcf89234 (table test-table [id=575f9c341b394fbcb6c1409de3d28dd1]) successfully deleted
I20260319 16:33:03.889469 27681 ts_tablet_manager.cc:1929] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:03.889516 27681 log.cc:1199] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/c16410dc2f9f42198defe229dcf89234
I20260319 16:33:03.889747 27681 ts_tablet_manager.cc:1950] T c16410dc2f9f42198defe229dcf89234 P cf1f4292aaeb45d78885afefb33d0a5f: Deleting consensus metadata
I20260319 16:33:03.890097 27070 catalog_manager.cc:5002] TS cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855): tablet c16410dc2f9f42198defe229dcf89234 (table test-table [id=575f9c341b394fbcb6c1409de3d28dd1]) successfully deleted
I20260319 16:33:03.890381 27070 catalog_manager.cc:5002] TS 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649): tablet c16410dc2f9f42198defe229dcf89234 (table test-table [id=575f9c341b394fbcb6c1409de3d28dd1]) successfully deleted
I20260319 16:33:03.890966 27367 tablet_service.cc:1511] Processing CreateTablet for tablet d26be00868cf4098b4f6206e5cb0a64c (DEFAULT_TABLE table=test-table [id=c9083b1b240840e29011b8c19d5c6f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:03.890967 27225 tablet_service.cc:1511] Processing CreateTablet for tablet d26be00868cf4098b4f6206e5cb0a64c (DEFAULT_TABLE table=test-table [id=c9083b1b240840e29011b8c19d5c6f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:03.891073 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26be00868cf4098b4f6206e5cb0a64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:03.891074 27367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26be00868cf4098b4f6206e5cb0a64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:03.892010 27691 tablet_bootstrap.cc:492] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: Bootstrap starting.
I20260319 16:33:03.892069 27509 tablet_service.cc:1511] Processing CreateTablet for tablet d26be00868cf4098b4f6206e5cb0a64c (DEFAULT_TABLE table=test-table [id=c9083b1b240840e29011b8c19d5c6f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:03.892164 27509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26be00868cf4098b4f6206e5cb0a64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:03.892346 27691 tablet_bootstrap.cc:654] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:03.892364 27692 tablet_bootstrap.cc:492] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327: Bootstrap starting.
I20260319 16:33:03.892696 27692 tablet_bootstrap.cc:654] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:03.892976 27691 tablet_bootstrap.cc:492] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: No bootstrap required, opened a new log
I20260319 16:33:03.893026 27691 ts_tablet_manager.cc:1403] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:03.893142 27691 raft_consensus.cc:359] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.893234 27691 raft_consensus.cc:385] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:03.893254 27691 raft_consensus.cc:740] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b815513b7f84c3fb82696f646ec8487, State: Initialized, Role: FOLLOWER
I20260319 16:33:03.893258 27692 tablet_bootstrap.cc:492] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327: No bootstrap required, opened a new log
I20260319 16:33:03.893294 27692 ts_tablet_manager.cc:1403] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:03.893297 27691 consensus_queue.cc:260] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.893335 27695 tablet_bootstrap.cc:492] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: Bootstrap starting.
I20260319 16:33:03.893383 27691 ts_tablet_manager.cc:1434] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:03.893393 27692 raft_consensus.cc:359] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.893455 27692 raft_consensus.cc:385] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:03.893473 27692 raft_consensus.cc:740] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d2e70f3f8d4ab0810c3db8f74e4327, State: Initialized, Role: FOLLOWER
I20260319 16:33:03.893512 27692 consensus_queue.cc:260] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.893590 27692 ts_tablet_manager.cc:1434] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:03.893767 27695 tablet_bootstrap.cc:654] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:03.894760 27695 tablet_bootstrap.cc:492] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: No bootstrap required, opened a new log
I20260319 16:33:03.894807 27695 ts_tablet_manager.cc:1403] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:03.894951 27695 raft_consensus.cc:359] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.895013 27695 raft_consensus.cc:385] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:03.895031 27695 raft_consensus.cc:740] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1f4292aaeb45d78885afefb33d0a5f, State: Initialized, Role: FOLLOWER
I20260319 16:33:03.895071 27695 consensus_queue.cc:260] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.895156 27695 ts_tablet_manager.cc:1434] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:03.941941 27713 raft_consensus.cc:493] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:03.942034 27713 raft_consensus.cc:515] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.942225 27713 leader_election.cc:290] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649), 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801)
I20260319 16:33:03.942431 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d26be00868cf4098b4f6206e5cb0a64c" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487" is_pre_election: true
I20260319 16:33:03.942454 27245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d26be00868cf4098b4f6206e5cb0a64c" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" is_pre_election: true
I20260319 16:33:03.942543 27245 raft_consensus.cc:2468] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 0.
I20260319 16:33:03.942541 27387 raft_consensus.cc:2468] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 0.
I20260319 16:33:03.942720 27462 leader_election.cc:304] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b815513b7f84c3fb82696f646ec8487, cf1f4292aaeb45d78885afefb33d0a5f; no voters:
I20260319 16:33:03.942809 27713 raft_consensus.cc:2804] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:03.942842 27713 raft_consensus.cc:493] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:03.942866 27713 raft_consensus.cc:3060] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:03.943559 27713 raft_consensus.cc:515] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.943801 27713 leader_election.cc:290] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 election: Requested vote from peers 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649), 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801)
I20260319 16:33:03.943826 27387 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d26be00868cf4098b4f6206e5cb0a64c" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b815513b7f84c3fb82696f646ec8487"
I20260319 16:33:03.943837 27245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d26be00868cf4098b4f6206e5cb0a64c" candidate_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327"
I20260319 16:33:03.943897 27387 raft_consensus.cc:3060] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:03.943925 27245 raft_consensus.cc:3060] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:03.944429 27387 raft_consensus.cc:2468] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 1.
I20260319 16:33:03.944615 27462 leader_election.cc:304] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b815513b7f84c3fb82696f646ec8487, cf1f4292aaeb45d78885afefb33d0a5f; no voters:
I20260319 16:33:03.944691 27713 raft_consensus.cc:2804] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:03.944757 27713 raft_consensus.cc:697] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 1 LEADER]: Becoming Leader. State: Replica: cf1f4292aaeb45d78885afefb33d0a5f, State: Running, Role: LEADER
I20260319 16:33:03.944765 27245 raft_consensus.cc:2468] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1f4292aaeb45d78885afefb33d0a5f in term 1.
I20260319 16:33:03.944832 27713 consensus_queue.cc:237] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } }
I20260319 16:33:03.945494 27084 catalog_manager.cc:5671] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f reported cstate change: term changed from 0 to 1, leader changed from <none> to cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195). New cstate: current_term: 1 leader_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf1f4292aaeb45d78885afefb33d0a5f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42855 } health_report { overall_health: HEALTHY } } }
I20260319 16:33:03.957285 27387 raft_consensus.cc:1275] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Refusing update from remote peer cf1f4292aaeb45d78885afefb33d0a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:03.957502 27245 raft_consensus.cc:1275] T d26be00868cf4098b4f6206e5cb0a64c P 47d2e70f3f8d4ab0810c3db8f74e4327 [term 1 FOLLOWER]: Refusing update from remote peer cf1f4292aaeb45d78885afefb33d0a5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:03.957603 27713 consensus_queue.cc:1048] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b815513b7f84c3fb82696f646ec8487" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:03.957767 27702 consensus_queue.cc:1048] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47d2e70f3f8d4ab0810c3db8f74e4327" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260319 16:33:03.962548 27663 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260319 16:33:05.481298 27084 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:05.481421 27084 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:05.482271 27084 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 91df8ebd2d4f4aaca509fd688adf73fb: Sending DeleteTablet for 3 replicas of tablet d26be00868cf4098b4f6206e5cb0a64c
I20260319 16:33:05.482529 27509 tablet_service.cc:1558] Processing DeleteTablet for tablet d26be00868cf4098b4f6206e5cb0a64c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:05 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:41540
I20260319 16:33:05.482530 27224 tablet_service.cc:1558] Processing DeleteTablet for tablet d26be00868cf4098b4f6206e5cb0a64c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:05 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:36306
I20260319 16:33:05.482692 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 27160
I20260319 16:33:05.482934 27367 tablet_service.cc:1558] Processing DeleteTablet for tablet d26be00868cf4098b4f6206e5cb0a64c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:05 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33462
I20260319 16:33:05.483366 27745 tablet_replica.cc:333] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: stopping tablet replica
I20260319 16:33:05.483425 27745 raft_consensus.cc:2243] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:05.483481 27745 raft_consensus.cc:2272] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:05.483706 27745 ts_tablet_manager.cc:1916] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:05.484166 27746 tablet_replica.cc:333] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: stopping tablet replica
I20260319 16:33:05.484225 27746 raft_consensus.cc:2243] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:05.484289 27746 raft_consensus.cc:2272] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:05.484505 27746 ts_tablet_manager.cc:1916] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:05.485473 27745 ts_tablet_manager.cc:1929] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260319 16:33:05.485530 27745 log.cc:1199] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/d26be00868cf4098b4f6206e5cb0a64c
I20260319 16:33:05.485786 27745 ts_tablet_manager.cc:1950] T d26be00868cf4098b4f6206e5cb0a64c P cf1f4292aaeb45d78885afefb33d0a5f: Deleting consensus metadata
I20260319 16:33:05.485787 27746 ts_tablet_manager.cc:1929] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260319 16:33:05.485836 27746 log.cc:1199] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/d26be00868cf4098b4f6206e5cb0a64c
I20260319 16:33:05.486052 27746 ts_tablet_manager.cc:1950] T d26be00868cf4098b4f6206e5cb0a64c P 7b815513b7f84c3fb82696f646ec8487: Deleting consensus metadata
I20260319 16:33:05.486111 27070 catalog_manager.cc:5002] TS cf1f4292aaeb45d78885afefb33d0a5f (127.24.187.195:42855): tablet d26be00868cf4098b4f6206e5cb0a64c (table test-table [id=c9083b1b240840e29011b8c19d5c6f98]) successfully deleted
I20260319 16:33:05.486380 27072 catalog_manager.cc:5002] TS 7b815513b7f84c3fb82696f646ec8487 (127.24.187.194:43801): tablet d26be00868cf4098b4f6206e5cb0a64c (table test-table [id=c9083b1b240840e29011b8c19d5c6f98]) successfully deleted
W20260319 16:33:05.489670 27070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:33649 (error 108)
I20260319 16:33:05.489926 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 27302
I20260319 16:33:05.490757 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:05.490061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33649 (local address 127.0.0.1:36330)
0319 16:33:05.490664 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:05.490716 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:33649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":59,"threads_started":1}
W20260319 16:33:05.490890 27070 catalog_manager.cc:4729] TS 47d2e70f3f8d4ab0810c3db8f74e4327 (127.24.187.193:33649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d26be00868cf4098b4f6206e5cb0a64c: Network error: Client connection negotiation failed: client connection to 127.24.187.193:33649: connect: Connection refused (error 111)
I20260319 16:33:05.495630 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 27444
I20260319 16:33:05.500413 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 27054
2026-03-19T16:33:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8788 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:05 dist-test-slave-svn8 krb5kdc[27751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:05 dist-test-slave-svn8 krb5kdc[27751](info): set up 2 sockets
Mar 19 16:33:05 dist-test-slave-svn8 krb5kdc[27751](info): commencing operation
krb5kdc: starting...
W20260319 16:33:07.350346 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.830s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:07 dist-test-slave-svn8 krb5kdc[27751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:07.484992 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42249
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:07.559703 27767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:07.559870 27767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:07.559891 27767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:07.561331 27767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:07.561370 27767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:07.561383 27767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:07.561393 27767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:07.561403 27767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:07.563041 27767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:07.563220 27767 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:07.563349 27767 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:33:07.565949 27767 server_base.cc:1060] running on GCE node
W20260319 16:33:07.565930 27772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:07.566042 27773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:07.565927 27775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:07.566339 27767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:07.566571 27767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:07.567772 27767 hybrid_clock.cc:648] HybridClock initialized: now 1773937987567747 us; error 31 us; skew 500 ppm
Mar 19 16:33:07 dist-test-slave-svn8 krb5kdc[27751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937987, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:07.570212 27767 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:07.570715 27767 webserver.cc:492] Webserver started at http://127.24.187.254:41953/ using document root <none> and password file <none>
I20260319 16:33:07.570874 27767 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:07.570909 27767 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:07.571007 27767 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:07.571909 27767 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "2d2887d67a6d463d9950b83e791edbeb"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "ba2dee015ea1f29f88f3cdac695a9749"
server_key_iv: "cc8975ea31402775ff270d2099c66d02"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.572191 27767 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "2d2887d67a6d463d9950b83e791edbeb"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "ba2dee015ea1f29f88f3cdac695a9749"
server_key_iv: "cc8975ea31402775ff270d2099c66d02"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.573568 27767 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:33:07.574502 27782 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:07.574769 27767 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:07.574846 27767 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2d2887d67a6d463d9950b83e791edbeb"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "ba2dee015ea1f29f88f3cdac695a9749"
server_key_iv: "cc8975ea31402775ff270d2099c66d02"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.574914 27767 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:07.591079 27767 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:07.596999 27767 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:07.597138 27767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:07.601235 27767 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:42249
I20260319 16:33:07.601279 27834 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:42249 every 8 connection(s)
I20260319 16:33:07.601567 27767 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:07.602126 27835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:07.604462 27835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb: Bootstrap starting.
I20260319 16:33:07.605026 27835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:07.605245 27835 log.cc:826] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:07.605950 27835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb: No bootstrap required, opened a new log
I20260319 16:33:07.607115 27835 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2887d67a6d463d9950b83e791edbeb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42249 } }
I20260319 16:33:07.607232 27835 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:07.607251 27835 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2887d67a6d463d9950b83e791edbeb, State: Initialized, Role: FOLLOWER
I20260319 16:33:07.607358 27835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2887d67a6d463d9950b83e791edbeb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42249 } }
I20260319 16:33:07.607417 27835 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:07.607441 27835 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:07.607481 27835 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:07.608008 27835 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2887d67a6d463d9950b83e791edbeb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42249 } }
I20260319 16:33:07.608099 27835 leader_election.cc:304] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d2887d67a6d463d9950b83e791edbeb; no voters:
I20260319 16:33:07.608230 27835 leader_election.cc:290] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:07.608284 27838 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:07.608436 27838 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [term 1 LEADER]: Becoming Leader. State: Replica: 2d2887d67a6d463d9950b83e791edbeb, State: Running, Role: LEADER
I20260319 16:33:07.608457 27835 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:07.608522 27838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2887d67a6d463d9950b83e791edbeb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42249 } }
I20260319 16:33:07.608850 27839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d2887d67a6d463d9950b83e791edbeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2887d67a6d463d9950b83e791edbeb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42249 } } }
I20260319 16:33:07.608965 27839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:07.608886 27840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2887d67a6d463d9950b83e791edbeb. Latest consensus state: current_term: 1 leader_uuid: "2d2887d67a6d463d9950b83e791edbeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2887d67a6d463d9950b83e791edbeb" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42249 } } }
I20260319 16:33:07.609040 27840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:07.609256 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 27767
I20260319 16:33:07.609349 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:07.609375 27845 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:07.609506 25327 external_mini_cluster.cc:1468] Setting key 9007c42b748bd8b5a2d9e7864370bd63
I20260319 16:33:07.610078 27845 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:07.611500 27845 catalog_manager.cc:1357] Generated new cluster ID: b15c3ccd7cdc4ec2a3493cd7fcf3a56c
I20260319 16:33:07.611548 27845 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:33:07 dist-test-slave-svn8 krb5kdc[27751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:07.619992 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:07.610095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.24.187.254:42249)
0319 16:33:07.610271 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:07.610352 (+ 81us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:07.610500 (+ 148us) server_negotiation.cc:408] Connection header received
0319 16:33:07.610666 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:07.610676 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:07.610731 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:07.610856 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:07.611733 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.612912 (+ 1179us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:07.613839 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.614073 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:07.616465 (+ 2392us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:07.616470 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:07.616471 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:07.616489 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:07.618664 (+ 2175us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:07.619034 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:07.619036 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:07.619037 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:07.619091 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:07.619229 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:07.619230 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:07.619230 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:07.619439 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:07.619550 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:07.619695 (+ 145us) server_negotiation.cc:300] Negotiation successful
0319 16:33:07.619811 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260319 16:33:07.625288 27845 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:07.625867 27845 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:07.632200 27845 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2d2887d67a6d463d9950b83e791edbeb: Generated new TSK 0
I20260319 16:33:07.632391 27845 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:07.684803 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42249
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:07.759888 27863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:07.760051 27863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:07.760083 27863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:07.761567 27863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:07.761616 27863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:07.761652 27863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:07.763293 27863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:07.763518 27863 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:07.763681 27863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:07.766060 27871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:07.766060 27869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:07.766171 27863 server_base.cc:1060] running on GCE node
W20260319 16:33:07.766057 27868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:07.766485 27863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:07.766705 27863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:07.767835 27863 hybrid_clock.cc:648] HybridClock initialized: now 1773937987767817 us; error 31 us; skew 500 ppm
Mar 19 16:33:07 dist-test-slave-svn8 krb5kdc[27751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937987, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:07.770231 27863 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:07.770772 27863 webserver.cc:492] Webserver started at http://127.24.187.193:42175/ using document root <none> and password file <none>
I20260319 16:33:07.770937 27863 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:07.770988 27863 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:07.771091 27863 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:07.771973 27863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "41a032e6603442eeb0ee7f25d99d6ef1"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "af97a60dbd07f627f8386994b6967f33"
server_key_iv: "4033b5695dc3de3d9802215b342b5479"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.772274 27863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "41a032e6603442eeb0ee7f25d99d6ef1"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "af97a60dbd07f627f8386994b6967f33"
server_key_iv: "4033b5695dc3de3d9802215b342b5479"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.773470 27863 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:33:07.774235 27878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:07.774459 27863 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:33:07.774521 27863 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "41a032e6603442eeb0ee7f25d99d6ef1"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "af97a60dbd07f627f8386994b6967f33"
server_key_iv: "4033b5695dc3de3d9802215b342b5479"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.774585 27863 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:07.798506 27863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:07.800236 27863 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:07.800352 27863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:07.800540 27863 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:07.800823 27863 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:07.800855 27863 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:07.800882 27863 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:07.800911 27863 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:07.806537 27863 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:36147
I20260319 16:33:07.806581 27991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:36147 every 8 connection(s)
I20260319 16:33:07.806849 27863 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:07.809612 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 27863
I20260319 16:33:07.809758 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:07.809937 25327 external_mini_cluster.cc:1468] Setting key 85bd8c27972ddc0dd21243be9cbc5519
Mar 19 16:33:07 dist-test-slave-svn8 krb5kdc[27751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937987, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:07.815339 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:07.807518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:44555 (local address 127.24.187.254:42249)
0319 16:33:07.807573 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:07.807574 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:07.807706 (+ 132us) server_negotiation.cc:408] Connection header received
0319 16:33:07.808195 (+ 489us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:07.808197 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:07.808212 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:07.808274 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:07.809096 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.809734 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:07.810424 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.810595 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:07.812942 (+ 2347us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:07.812946 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:07.812946 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:07.812953 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:07.814244 (+ 1291us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:07.814592 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:07.814593 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:07.814593 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:07.814615 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:07.814775 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:07.814776 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:07.814777 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:07.814862 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:07.814916 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:07.815148 (+ 232us) server_negotiation.cc:300] Negotiation successful
0319 16:33:07.815242 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260319 16:33:07.815410 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:07.807464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42249 (local address 127.24.187.193:44555)
0319 16:33:07.807597 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:07.807628 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:07.808099 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:07.808367 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:07.808371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:07.808698 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:07.808991 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:07.808995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.809822 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:07.809824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:07.810310 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:07.810313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.810438 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:07.811315 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:07.811329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:07.812861 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:07.814318 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:07.814324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:07.814326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:07.814527 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:07.814672 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:07.814673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:07.814673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:07.814727 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:07.814913 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:07.814918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:07.815025 (+ 107us) client_negotiation.cc:769] Sending connection context
0319 16:33:07.815121 (+ 96us) client_negotiation.cc:240] Negotiation successful
0319 16:33:07.815246 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260319 16:33:07.816291 27992 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42249
I20260319 16:33:07.816372 27992 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:07.816550 27992 heartbeater.cc:507] Master 127.24.187.254:42249 requested a full tablet report, sending...
I20260319 16:33:07.817010 27799 ts_manager.cc:194] Registered new tserver with Master: 41a032e6603442eeb0ee7f25d99d6ef1 (127.24.187.193:36147)
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
I20260319 16:33:07.817672 27799 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:44555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:07.856005 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42249
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:07.930128 27999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:07.930282 27999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:07.930303 27999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:07.931726 27999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:07.931767 27999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:07.931792 27999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:07.933365 27999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:07.933552 27999 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:07.933727 27999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:07.936105 28004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:07.936199 28007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:07.936122 27999 server_base.cc:1060] running on GCE node
W20260319 16:33:07.936151 28005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:07.936503 27999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:07.936682 27999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:07.937788 27999 hybrid_clock.cc:648] HybridClock initialized: now 1773937987937758 us; error 40 us; skew 500 ppm
Mar 19 16:33:07 dist-test-slave-svn8 krb5kdc[27751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937987, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:07.940121 27999 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:07.940685 27999 webserver.cc:492] Webserver started at http://127.24.187.194:39393/ using document root <none> and password file <none>
I20260319 16:33:07.940846 27999 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:07.940881 27999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:07.940977 27999 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:07.941926 27999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "e534ef535755416d988f3a7fa7c734f7"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "d473014e0818f6eb3895a52cabdcbeeb"
server_key_iv: "e2f985bb6a8ae6f516a5585929b52184"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.942214 27999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "e534ef535755416d988f3a7fa7c734f7"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "d473014e0818f6eb3895a52cabdcbeeb"
server_key_iv: "e2f985bb6a8ae6f516a5585929b52184"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.943286 27999 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:07.943961 28014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:07.944149 27999 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:07.944207 27999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "e534ef535755416d988f3a7fa7c734f7"
format_stamp: "Formatted at 2026-03-19 16:33:07 on dist-test-slave-svn8"
server_key: "d473014e0818f6eb3895a52cabdcbeeb"
server_key_iv: "e2f985bb6a8ae6f516a5585929b52184"
server_key_version: "encryptionkey@0"
I20260319 16:33:07.944268 27999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:07.956245 27999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:07.958213 27999 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:07.958335 27999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:07.958521 27999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:07.958837 27999 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:07.958870 27999 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:07.958899 27999 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:07.958925 27999 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:07.964262 27999 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:36021
I20260319 16:33:07.964362 28127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:36021 every 8 connection(s)
I20260319 16:33:07.964612 27999 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:33:07 dist-test-slave-svn8 krb5kdc[27751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937987, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:07.970374 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 27999
I20260319 16:33:07.970466 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:07.970652 25327 external_mini_cluster.cc:1468] Setting key fe592b642232dcc112bf8f0681f694c1
I20260319 16:33:07.972699 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:07.965293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:34367 (local address 127.24.187.254:42249)
0319 16:33:07.965356 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:07.965358 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:07.965493 (+ 135us) server_negotiation.cc:408] Connection header received
0319 16:33:07.966026 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:07.966028 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:07.966037 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:07.966103 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:07.966801 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.967295 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:07.967916 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.968042 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:07.970256 (+ 2214us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:07.970260 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:07.970260 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:07.970266 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:07.971679 (+ 1413us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:07.972006 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:07.972007 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:07.972008 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:07.972028 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:07.972176 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:07.972176 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:07.972177 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:07.972260 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:07.972313 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:07.972536 (+ 223us) server_negotiation.cc:300] Negotiation successful
0319 16:33:07.972615 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260319 16:33:07.972824 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:07.965246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42249 (local address 127.24.187.194:34367)
0319 16:33:07.965391 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:07.965422 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:07.965922 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:07.966145 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:07.966148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:07.966442 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:07.966702 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:07.966706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.967402 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:07.967404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:07.967825 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:07.967826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:07.967942 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:07.968778 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:07.968792 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:07.970167 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:07.971750 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:07.971755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:07.971757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:07.971946 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:07.972076 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:07.972077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:07.972077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:07.972128 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:07.972310 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:07.972314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:07.972423 (+ 109us) client_negotiation.cc:769] Sending connection context
0319 16:33:07.972511 (+ 88us) client_negotiation.cc:240] Negotiation successful
0319 16:33:07.972631 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260319 16:33:07.973323 28128 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42249
I20260319 16:33:07.973413 28128 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:07.973664 28128 heartbeater.cc:507] Master 127.24.187.254:42249 requested a full tablet report, sending...
I20260319 16:33:07.974077 27799 ts_manager.cc:194] Registered new tserver with Master: e534ef535755416d988f3a7fa7c734f7 (127.24.187.194:36021)
I20260319 16:33:07.974562 27799 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:34367
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:08.014766 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42249
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:08.087965 28135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:08.088112 28135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:08.088131 28135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:08.089596 28135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:08.089638 28135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:08.089661 28135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:08.091238 28135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:08.091403 28135 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:08.091524 28135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:08.093811 28140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:08.093852 28141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:08.093852 28143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:08.094038 28135 server_base.cc:1060] running on GCE node
I20260319 16:33:08.094218 28135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:08.094403 28135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:08.095546 28135 hybrid_clock.cc:648] HybridClock initialized: now 1773937988095527 us; error 29 us; skew 500 ppm
Mar 19 16:33:08 dist-test-slave-svn8 krb5kdc[27751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937988, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:08.097949 28135 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:08.098415 28135 webserver.cc:492] Webserver started at http://127.24.187.195:36163/ using document root <none> and password file <none>
I20260319 16:33:08.098572 28135 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:08.098605 28135 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:08.098709 28135 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:08.099581 28135 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "2d9ea6545e75472ea38d9eada8c823fc"
format_stamp: "Formatted at 2026-03-19 16:33:08 on dist-test-slave-svn8"
server_key: "d08cabdb892206187b925c8bb1ba4c25"
server_key_iv: "a94b7a00ce100dafe9b8907dded72edc"
server_key_version: "encryptionkey@0"
I20260319 16:33:08.099874 28135 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "2d9ea6545e75472ea38d9eada8c823fc"
format_stamp: "Formatted at 2026-03-19 16:33:08 on dist-test-slave-svn8"
server_key: "d08cabdb892206187b925c8bb1ba4c25"
server_key_iv: "a94b7a00ce100dafe9b8907dded72edc"
server_key_version: "encryptionkey@0"
I20260319 16:33:08.100965 28135 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:08.101624 28150 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:08.101835 28135 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:08.101910 28135 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "2d9ea6545e75472ea38d9eada8c823fc"
format_stamp: "Formatted at 2026-03-19 16:33:08 on dist-test-slave-svn8"
server_key: "d08cabdb892206187b925c8bb1ba4c25"
server_key_iv: "a94b7a00ce100dafe9b8907dded72edc"
server_key_version: "encryptionkey@0"
I20260319 16:33:08.101986 28135 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:08.109990 28135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:08.111505 28135 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:08.111599 28135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:08.111740 28135 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:08.112006 28135 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:08.112035 28135 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:08.112053 28135 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:08.112075 28135 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:08.117147 28135 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:46791
I20260319 16:33:08.117205 28263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:46791 every 8 connection(s)
I20260319 16:33:08.117430 28135 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:33:08.118821 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 28135
I20260319 16:33:08.118914 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:08.119086 25327 external_mini_cluster.cc:1468] Setting key faa681f1a3082c3251b876a19b90660f
Mar 19 16:33:08 dist-test-slave-svn8 krb5kdc[27751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937988, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:08.126452 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:08.118134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:48371 (local address 127.24.187.254:42249)
0319 16:33:08.118191 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:08.118193 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:08.118229 (+ 36us) server_negotiation.cc:408] Connection header received
0319 16:33:08.118720 (+ 491us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:08.118723 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:08.118738 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:08.118840 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:08.119660 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:08.120454 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:08.121056 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:08.121180 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:08.123716 (+ 2536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:08.123721 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:08.123721 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:08.123728 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:08.125027 (+ 1299us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:08.125452 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:08.125454 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:08.125454 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:08.125483 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:08.125768 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:08.125769 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:08.125769 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:08.125872 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:08.125933 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:08.126263 (+ 330us) server_negotiation.cc:300] Negotiation successful
0319 16:33:08.126356 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260319 16:33:08.126472 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:08.118010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42249 (local address 127.24.187.195:48371)
0319 16:33:08.118139 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:08.118169 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:08.118622 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:08.118882 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:08.118886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:08.119213 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:08.119580 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:08.119585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:08.120560 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:08.120561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:08.120967 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:08.120969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:08.121070 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:08.122016 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:08.122031 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:08.123540 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:08.125114 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:08.125118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:08.125120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:08.125321 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:08.125573 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:08.125574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:08.125575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:08.125656 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:08.125954 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:08.125959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:08.126090 (+ 131us) client_negotiation.cc:769] Sending connection context
0319 16:33:08.126201 (+ 111us) client_negotiation.cc:240] Negotiation successful
0319 16:33:08.126315 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260319 16:33:08.126953 28264 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42249
I20260319 16:33:08.127041 28264 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:08.127344 28264 heartbeater.cc:507] Master 127.24.187.254:42249 requested a full tablet report, sending...
I20260319 16:33:08.127841 27799 ts_manager.cc:194] Registered new tserver with Master: 2d9ea6545e75472ea38d9eada8c823fc (127.24.187.195:46791)
I20260319 16:33:08.128230 27799 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:48371
I20260319 16:33:08.131273 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260319 16:33:08.142380 27856 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:08.141410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.24.187.254:42249)
0319 16:33:08.141498 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:08.141500 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:08.141594 (+ 94us) server_negotiation.cc:408] Connection header received
0319 16:33:08.141665 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:08.141666 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:08.141678 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:08.141791 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:08.142310 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44602: BlockingRecv error: recv got EOF from 127.0.0.1:44602 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260319 16:33:08.143558 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 27863
I20260319 16:33:08.148371 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 27999
I20260319 16:33:08.152503 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 28135
I20260319 16:33:08.156517 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 27767
2026-03-19T16:33:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2654 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:08 dist-test-slave-svn8 krb5kdc[28276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:08 dist-test-slave-svn8 krb5kdc[28276](info): set up 2 sockets
Mar 19 16:33:08 dist-test-slave-svn8 krb5kdc[28276](info): commencing operation
krb5kdc: starting...
W20260319 16:33:09.986408 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.811s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:10.120982 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36445
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:36445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:10.195533 28292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:10.195730 28292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:10.195755 28292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:10.197266 28292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:10.197310 28292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:10.197326 28292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:10.197337 28292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:10.197348 28292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:10.198999 28292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:36445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:10.199193 28292 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:10.199360 28292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:10.201669 28298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.201666 28300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.201916 28297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:10.201927 28292 server_base.cc:1060] running on GCE node
I20260319 16:33:10.202142 28292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:10.202355 28292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:10.203541 28292 hybrid_clock.cc:648] HybridClock initialized: now 1773937990203524 us; error 28 us; skew 500 ppm
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:10.205868 28292 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:10.206369 28292 webserver.cc:492] Webserver started at http://127.24.187.254:41341/ using document root <none> and password file <none>
I20260319 16:33:10.206530 28292 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:10.206564 28292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:10.206660 28292 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:10.207559 28292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "ae9159a351f14e0ea57b345afdcb1da7"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "17356cf900f50ac151f09eab07a301d8"
server_key_iv: "de18ced1092675fc750acedd7ab00612"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.207844 28292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "ae9159a351f14e0ea57b345afdcb1da7"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "17356cf900f50ac151f09eab07a301d8"
server_key_iv: "de18ced1092675fc750acedd7ab00612"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.208942 28292 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:33:10.209566 28307 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.209792 28292 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.209859 28292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "ae9159a351f14e0ea57b345afdcb1da7"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "17356cf900f50ac151f09eab07a301d8"
server_key_iv: "de18ced1092675fc750acedd7ab00612"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.209923 28292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:10.224013 28292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:10.225994 28292 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:10.226128 28292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:10.230474 28292 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:36445
I20260319 16:33:10.230558 28359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:36445 every 8 connection(s)
I20260319 16:33:10.230954 28292 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:10.231565 28360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:10.233954 28360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7: Bootstrap starting.
I20260319 16:33:10.234560 28360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:10.234805 28360 log.cc:826] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:10.235277 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 28292
I20260319 16:33:10.235373 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:10.235540 25327 external_mini_cluster.cc:1468] Setting key 3d1f46d32adf20eb7bdab4812d892bf2
I20260319 16:33:10.235580 28360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7: No bootstrap required, opened a new log
I20260319 16:33:10.237218 28360 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9159a351f14e0ea57b345afdcb1da7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36445 } }
I20260319 16:33:10.237340 28360 raft_consensus.cc:385] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:10.237366 28360 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9159a351f14e0ea57b345afdcb1da7, State: Initialized, Role: FOLLOWER
I20260319 16:33:10.237488 28360 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9159a351f14e0ea57b345afdcb1da7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36445 } }
I20260319 16:33:10.237556 28360 raft_consensus.cc:399] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:10.237584 28360 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:10.237622 28360 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:10.238273 28360 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9159a351f14e0ea57b345afdcb1da7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36445 } }
I20260319 16:33:10.238379 28360 leader_election.cc:304] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae9159a351f14e0ea57b345afdcb1da7; no voters:
I20260319 16:33:10.238525 28360 leader_election.cc:290] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:10.238565 28365 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:10.238688 28365 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [term 1 LEADER]: Becoming Leader. State: Replica: ae9159a351f14e0ea57b345afdcb1da7, State: Running, Role: LEADER
I20260319 16:33:10.238734 28360 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:10.238927 28365 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9159a351f14e0ea57b345afdcb1da7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36445 } }
I20260319 16:33:10.239224 28365 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae9159a351f14e0ea57b345afdcb1da7. Latest consensus state: current_term: 1 leader_uuid: "ae9159a351f14e0ea57b345afdcb1da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9159a351f14e0ea57b345afdcb1da7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36445 } } }
I20260319 16:33:10.239302 28365 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:10.239233 28367 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae9159a351f14e0ea57b345afdcb1da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9159a351f14e0ea57b345afdcb1da7" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36445 } } }
I20260319 16:33:10.239475 28367 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7 [sys.catalog]: This master's current role is: LEADER
W20260319 16:33:10.241050 28380 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:33:10.241101 28380 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:10.242343 28369 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:10.242537 28369 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:10.244019 28369 catalog_manager.cc:1357] Generated new cluster ID: d6b2205968ca4b29ace21997421f405c
I20260319 16:33:10.244073 28369 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:10.245648 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:10.236121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.24.187.254:36445)
0319 16:33:10.236304 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.236320 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.236343 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:33:10.236720 (+ 377us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.236727 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.236788 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.236900 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.237938 (+ 1038us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.239100 (+ 1162us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:10.241330 (+ 2230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.241577 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:10.242445 (+ 868us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:10.242453 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:10.242453 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:10.242474 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:10.244345 (+ 1871us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.244650 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.244652 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.244652 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.244695 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.244930 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.244931 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.244932 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.245130 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:10.245250 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:10.245373 (+ 123us) server_negotiation.cc:300] Negotiation successful
0319 16:33:10.245490 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260319 16:33:10.256709 28369 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:10.257412 28369 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:10.271399 28369 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ae9159a351f14e0ea57b345afdcb1da7: Generated new TSK 0
I20260319 16:33:10.271606 28369 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:10.318578 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36445
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:10.392504 28388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:10.392679 28388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:10.392710 28388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:10.394182 28388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:10.394222 28388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:10.394256 28388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:10.395856 28388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:10.396071 28388 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:10.396225 28388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:10.398631 28394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.398665 28393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.398644 28396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:10.398739 28388 server_base.cc:1060] running on GCE node
I20260319 16:33:10.399009 28388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:10.399171 28388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:10.400312 28388 hybrid_clock.cc:648] HybridClock initialized: now 1773937990400293 us; error 30 us; skew 500 ppm
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:10.402595 28388 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:10.403075 28388 webserver.cc:492] Webserver started at http://127.24.187.193:37337/ using document root <none> and password file <none>
I20260319 16:33:10.403230 28388 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:10.403263 28388 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:10.403362 28388 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:10.404245 28388 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "8b607ea0fdd548e3ba38534ee398d8fa"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "306fc5b459735e3b87f5270b20ae6fc2"
server_key_iv: "513cec6a585b85bd43eae30536575f6a"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.404523 28388 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "8b607ea0fdd548e3ba38534ee398d8fa"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "306fc5b459735e3b87f5270b20ae6fc2"
server_key_iv: "513cec6a585b85bd43eae30536575f6a"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.405632 28388 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:10.406343 28403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.406514 28388 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.406577 28388 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "8b607ea0fdd548e3ba38534ee398d8fa"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "306fc5b459735e3b87f5270b20ae6fc2"
server_key_iv: "513cec6a585b85bd43eae30536575f6a"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.406638 28388 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:10.421031 28388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:10.422871 28388 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:10.422973 28388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:10.423178 28388 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:10.423475 28388 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:10.423506 28388 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.423537 28388 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:10.423553 28388 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.428979 28388 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:44285
I20260319 16:33:10.429071 28516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:44285 every 8 connection(s)
I20260319 16:33:10.429303 28388 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:10.432940 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 28388
I20260319 16:33:10.433036 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:10.433182 25327 external_mini_cluster.cc:1468] Setting key 1a45ef9e73597411addf0d210a8445e8
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:10.438618 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:10.430030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:47421 (local address 127.24.187.254:36445)
0319 16:33:10.430097 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.430099 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.430216 (+ 117us) server_negotiation.cc:408] Connection header received
0319 16:33:10.430772 (+ 556us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.430774 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.430785 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.430840 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.431622 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.432106 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:10.432734 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.432899 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:10.435435 (+ 2536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:10.435440 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:10.435440 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:10.435464 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:10.437242 (+ 1778us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.437744 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.437745 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.437746 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.437774 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.437989 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.437990 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.437991 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.438079 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:10.438128 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:10.438456 (+ 328us) server_negotiation.cc:300] Negotiation successful
0319 16:33:10.438541 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260319 16:33:10.438767 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:10.429957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36445 (local address 127.24.187.193:47421)
0319 16:33:10.430125 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:10.430153 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:10.430662 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:10.430920 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:10.430925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:10.431220 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:10.431505 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:10.431509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.432206 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:10.432207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:10.432646 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:10.432649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.432773 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:10.433285 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:10.433301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:10.435279 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:10.437340 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:10.437346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:10.437349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:10.437624 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:10.437840 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:10.437840 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:10.437841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:10.437922 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:10.438135 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:10.438140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:10.438290 (+ 150us) client_negotiation.cc:769] Sending connection context
0319 16:33:10.438421 (+ 131us) client_negotiation.cc:240] Negotiation successful
0319 16:33:10.438565 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260319 16:33:10.439286 28517 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36445
I20260319 16:33:10.439388 28517 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:10.439622 28517 heartbeater.cc:507] Master 127.24.187.254:36445 requested a full tablet report, sending...
I20260319 16:33:10.440152 28324 ts_manager.cc:194] Registered new tserver with Master: 8b607ea0fdd548e3ba38534ee398d8fa (127.24.187.193:44285)
I20260319 16:33:10.440837 28324 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:47421
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:10.480715 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36445
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:10.557437 28524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:10.557611 28524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:10.557641 28524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:10.559180 28524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:10.559229 28524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:10.559265 28524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:10.560988 28524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:10.561211 28524 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:10.561393 28524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:33:10.564020 28524 server_base.cc:1060] running on GCE node
W20260319 16:33:10.563966 28530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.563966 28532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.563966 28529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:10.564406 28524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:10.564618 28524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:10.565773 28524 hybrid_clock.cc:648] HybridClock initialized: now 1773937990565746 us; error 37 us; skew 500 ppm
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:10.568219 28524 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:10.568719 28524 webserver.cc:492] Webserver started at http://127.24.187.194:33817/ using document root <none> and password file <none>
I20260319 16:33:10.568879 28524 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:10.568912 28524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:10.569010 28524 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:10.569948 28524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "000f3f2e68294b47a543ef6582eed7ce"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "0c38e76a422a9ee8f202a5bf00451d2f"
server_key_iv: "99d0dbeef8537ac62e70ac96a33af172"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.570230 28524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "000f3f2e68294b47a543ef6582eed7ce"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "0c38e76a422a9ee8f202a5bf00451d2f"
server_key_iv: "99d0dbeef8537ac62e70ac96a33af172"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.571385 28524 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:10.572104 28539 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.572352 28524 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:10.572422 28524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "000f3f2e68294b47a543ef6582eed7ce"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "0c38e76a422a9ee8f202a5bf00451d2f"
server_key_iv: "99d0dbeef8537ac62e70ac96a33af172"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.572485 28524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:10.590869 28524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:10.592908 28524 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:10.593029 28524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:10.593264 28524 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:10.593593 28524 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:10.593626 28524 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.593665 28524 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:10.593683 28524 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.599283 28524 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:35805
I20260319 16:33:10.599362 28652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:35805 every 8 connection(s)
I20260319 16:33:10.599614 28524 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:10.605517 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 28524
I20260319 16:33:10.605620 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:10.605827 25327 external_mini_cluster.cc:1468] Setting key 2612cd406800b4c2d8288f952a6f3705
I20260319 16:33:10.608644 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:10.600225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:40089 (local address 127.24.187.254:36445)
0319 16:33:10.600266 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.600267 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.600553 (+ 286us) server_negotiation.cc:408] Connection header received
0319 16:33:10.601086 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.601088 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.601098 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.601156 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.601951 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.602436 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:10.602974 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.603097 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:10.605507 (+ 2410us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:10.605512 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:10.605512 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:10.605521 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:10.607288 (+ 1767us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.607702 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.607703 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.607704 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.607731 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.607938 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.607938 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.607939 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.608062 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:10.608129 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:10.608492 (+ 363us) server_negotiation.cc:300] Negotiation successful
0319 16:33:10.608567 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260319 16:33:10.608829 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:10.600182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36445 (local address 127.24.187.194:40089)
0319 16:33:10.600327 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:10.600359 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:10.600985 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:10.601203 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:10.601207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:10.601537 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:10.601855 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:10.601860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.602526 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:10.602527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:10.602899 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:10.602901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.602998 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:10.603477 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:10.603494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:10.605416 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:10.607382 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:10.607388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:10.607390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:10.607629 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:10.607810 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:10.607811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:10.607812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:10.607880 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:10.608153 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:10.608160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:10.608330 (+ 170us) client_negotiation.cc:769] Sending connection context
0319 16:33:10.608463 (+ 133us) client_negotiation.cc:240] Negotiation successful
0319 16:33:10.608628 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260319 16:33:10.609258 28653 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36445
I20260319 16:33:10.609337 28653 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:10.609535 28653 heartbeater.cc:507] Master 127.24.187.254:36445 requested a full tablet report, sending...
I20260319 16:33:10.609935 28324 ts_manager.cc:194] Registered new tserver with Master: 000f3f2e68294b47a543ef6582eed7ce (127.24.187.194:35805)
I20260319 16:33:10.610316 28324 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:40089
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:10.651993 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36445
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:10.726116 28660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:10.726290 28660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:10.726315 28660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:10.727811 28660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:10.727859 28660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:10.727883 28660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:10.729549 28660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:10.729779 28660 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:10.729951 28660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:10.732296 28666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.732415 28665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:10.732323 28668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:10.732625 28660 server_base.cc:1060] running on GCE node
I20260319 16:33:10.732790 28660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:10.733009 28660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:10.734192 28660 hybrid_clock.cc:648] HybridClock initialized: now 1773937990734172 us; error 36 us; skew 500 ppm
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:10.736773 28660 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:10.737293 28660 webserver.cc:492] Webserver started at http://127.24.187.195:40617/ using document root <none> and password file <none>
I20260319 16:33:10.737510 28660 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:10.737546 28660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:10.737646 28660 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:10.738557 28660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "20eb5f63a6334ceda0857df039fec894"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "64845838705fe74906612be444164161"
server_key_iv: "4b8870be56c794736698b10e066ca686"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.738895 28660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "20eb5f63a6334ceda0857df039fec894"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "64845838705fe74906612be444164161"
server_key_iv: "4b8870be56c794736698b10e066ca686"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.740121 28660 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:10.740898 28675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.741048 28660 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:10.741110 28660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "20eb5f63a6334ceda0857df039fec894"
format_stamp: "Formatted at 2026-03-19 16:33:10 on dist-test-slave-svn8"
server_key: "64845838705fe74906612be444164161"
server_key_iv: "4b8870be56c794736698b10e066ca686"
server_key_version: "encryptionkey@0"
I20260319 16:33:10.741174 28660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:10.753171 28660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:10.754938 28660 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:10.755064 28660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:10.755237 28660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:10.755499 28660 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:10.755527 28660 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.755551 28660 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:10.755568 28660 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:10.760767 28660 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:36765
I20260319 16:33:10.760839 28789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:36765 every 8 connection(s)
I20260319 16:33:10.761071 28660 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:10.766558 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 28660
I20260319 16:33:10.766641 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:10.766810 25327 external_mini_cluster.cc:1468] Setting key 4eae72125a75cd632c4b01ce6e3c6b4b
I20260319 16:33:10.769217 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:10.761792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:36359 (local address 127.24.187.254:36445)
0319 16:33:10.761848 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.761851 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.762000 (+ 149us) server_negotiation.cc:408] Connection header received
0319 16:33:10.762532 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.762533 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.762568 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.762616 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.763331 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.763819 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:10.764324 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.764493 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:10.766802 (+ 2309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:10.766806 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:10.766806 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:10.766813 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:10.768295 (+ 1482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.768632 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.768633 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.768633 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.768650 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:10.768777 (+ 127us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:10.768778 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:10.768778 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:10.768858 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:10.768904 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:10.769089 (+ 185us) server_negotiation.cc:300] Negotiation successful
0319 16:33:10.769154 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:33:10.769312 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:10.761758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36445 (local address 127.24.187.195:36359)
0319 16:33:10.761900 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:10.761926 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:10.762438 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:10.762701 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:10.762704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:10.762978 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:10.763242 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:10.763245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.763892 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:10.763893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:10.764255 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:10.764256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:10.764361 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:10.764821 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:10.764835 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:10.766718 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:10.768377 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:10.768383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:10.768384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:10.768573 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:10.768695 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:10.768695 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:10.768696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:10.768740 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:10.768897 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:10.768899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:10.768991 (+ 92us) client_negotiation.cc:769] Sending connection context
0319 16:33:10.769066 (+ 75us) client_negotiation.cc:240] Negotiation successful
0319 16:33:10.769159 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260319 16:33:10.769748 28790 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36445
I20260319 16:33:10.769836 28790 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:10.770012 28790 heartbeater.cc:507] Master 127.24.187.254:36445 requested a full tablet report, sending...
I20260319 16:33:10.770372 28324 ts_manager.cc:194] Registered new tserver with Master: 20eb5f63a6334ceda0857df039fec894 (127.24.187.195:36765)
I20260319 16:33:10.770733 28324 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:36359
I20260319 16:33:10.778755 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260319 16:33:10.869287 28364 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:10.867651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.24.187.254:36445)
0319 16:33:10.867709 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.867711 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.867881 (+ 170us) server_negotiation.cc:408] Connection header received
0319 16:33:10.868284 (+ 403us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.868286 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.868298 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.868499 (+ 201us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.869242 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59158: BlockingRecv error: recv got EOF from 127.0.0.1:59158 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260319 16:33:10.870055 28364 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:10.869607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.24.187.254:36445)
0319 16:33:10.869683 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.869685 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.869693 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:10.869730 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.869731 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.869739 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.869784 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.870013 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59172: BlockingRecv error: recv got EOF from 127.0.0.1:59172 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260319 16:33:10.870647 28364 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:10.870275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.24.187.254:36445)
0319 16:33:10.870307 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.870308 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.870314 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:33:10.870329 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.870329 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.870336 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.870371 (+ 35us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.870617 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59188: BlockingRecv error: recv got EOF from 127.0.0.1:59188 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260319 16:33:10.872232 28364 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:10.871714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.24.187.254:36445)
0319 16:33:10.871794 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:10.871795 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:10.871804 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:10.871820 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:10.871820 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:10.871828 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:10.871878 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:10.872190 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59198: BlockingRecv error: recv got EOF from 127.0.0.1:59198 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260319 16:33:10.877513 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 28388
I20260319 16:33:10.881676 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 28524
I20260319 16:33:10.885486 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 28660
I20260319 16:33:10.889112 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 28292
2026-03-19T16:33:10Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (2732 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28813](info): set up 2 sockets
Mar 19 16:33:10 dist-test-slave-svn8 krb5kdc[28813](info): commencing operation
krb5kdc: starting...
W20260319 16:33:12.722532 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:12 dist-test-slave-svn8 krb5kdc[28813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:12.856583 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37055
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:12.929401 28829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:12.929569 28829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:12.929589 28829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:12.931002 28829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:12.931043 28829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:12.931056 28829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:12.931066 28829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:12.931077 28829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:12.932595 28829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:37055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:12.932757 28829 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:12.932888 28829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:12.935107 28834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:12.935184 28837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:12.935281 28829 server_base.cc:1060] running on GCE node
W20260319 16:33:12.935209 28835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:12.935637 28829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:12.935823 28829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:12.936955 28829 hybrid_clock.cc:648] HybridClock initialized: now 1773937992936940 us; error 26 us; skew 500 ppm
Mar 19 16:33:12 dist-test-slave-svn8 krb5kdc[28813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937992, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:12.939193 28829 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:12.939708 28829 webserver.cc:492] Webserver started at http://127.24.187.254:36641/ using document root <none> and password file <none>
I20260319 16:33:12.939862 28829 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:12.939895 28829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:12.939991 28829 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:12.940866 28829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "9773d73805c940cab4788dd01eea2750"
format_stamp: "Formatted at 2026-03-19 16:33:12 on dist-test-slave-svn8"
server_key: "81f2cf896cff71811f327d160befa294"
server_key_iv: "d2814cc44c8dbcfedb6c7eeb1629d8fa"
server_key_version: "encryptionkey@0"
I20260319 16:33:12.941169 28829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "9773d73805c940cab4788dd01eea2750"
format_stamp: "Formatted at 2026-03-19 16:33:12 on dist-test-slave-svn8"
server_key: "81f2cf896cff71811f327d160befa294"
server_key_iv: "d2814cc44c8dbcfedb6c7eeb1629d8fa"
server_key_version: "encryptionkey@0"
I20260319 16:33:12.942281 28829 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:33:12.943038 28844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:12.943300 28829 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:12.943378 28829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "9773d73805c940cab4788dd01eea2750"
format_stamp: "Formatted at 2026-03-19 16:33:12 on dist-test-slave-svn8"
server_key: "81f2cf896cff71811f327d160befa294"
server_key_iv: "d2814cc44c8dbcfedb6c7eeb1629d8fa"
server_key_version: "encryptionkey@0"
I20260319 16:33:12.943485 28829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:12.953963 28829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:12.959515 28829 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:12.959650 28829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:12.963498 28829 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:37055
I20260319 16:33:12.963532 28896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:37055 every 8 connection(s)
I20260319 16:33:12.963824 28829 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:12.964476 28897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:12.966876 28897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750: Bootstrap starting.
I20260319 16:33:12.967393 28897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:12.967583 28897 log.cc:826] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:12.968209 28897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750: No bootstrap required, opened a new log
I20260319 16:33:12.969404 28897 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9773d73805c940cab4788dd01eea2750" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37055 } }
I20260319 16:33:12.969518 28897 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:12.969537 28897 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9773d73805c940cab4788dd01eea2750, State: Initialized, Role: FOLLOWER
I20260319 16:33:12.969627 28897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9773d73805c940cab4788dd01eea2750" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37055 } }
I20260319 16:33:12.969672 28897 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:12.969691 28897 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:12.969743 28897 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:12.970238 28897 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9773d73805c940cab4788dd01eea2750" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37055 } }
I20260319 16:33:12.970323 28897 leader_election.cc:304] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9773d73805c940cab4788dd01eea2750; no voters:
I20260319 16:33:12.970440 28897 leader_election.cc:290] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:12.970507 28900 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:12.970633 28900 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [term 1 LEADER]: Becoming Leader. State: Replica: 9773d73805c940cab4788dd01eea2750, State: Running, Role: LEADER
I20260319 16:33:12.970636 28897 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:12.970738 28900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9773d73805c940cab4788dd01eea2750" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37055 } }
I20260319 16:33:12.970963 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 28829
I20260319 16:33:12.971050 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:12.971048 28901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9773d73805c940cab4788dd01eea2750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9773d73805c940cab4788dd01eea2750" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37055 } } }
I20260319 16:33:12.971066 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9773d73805c940cab4788dd01eea2750. Latest consensus state: current_term: 1 leader_uuid: "9773d73805c940cab4788dd01eea2750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9773d73805c940cab4788dd01eea2750" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37055 } } }
I20260319 16:33:12.971124 28901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:12.971131 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:12.971242 25327 external_mini_cluster.cc:1468] Setting key abd8e5a346d55bab3518573c21c588be
I20260319 16:33:12.971307 28906 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:12.972465 28906 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:12.973918 28906 catalog_manager.cc:1357] Generated new cluster ID: aa6e36e967ba40cc855784e21822e3a0
I20260319 16:33:12.974959 28906 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:33:12 dist-test-slave-svn8 krb5kdc[28813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:12.980126 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:12.971701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.24.187.254:37055)
0319 16:33:12.971852 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:12.971864 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:12.972141 (+ 277us) server_negotiation.cc:408] Connection header received
0319 16:33:12.972269 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:12.972276 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:12.972323 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:12.972458 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:12.973544 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:12.974712 (+ 1168us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:12.975358 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:12.975588 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:12.977405 (+ 1817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:12.977432 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:12.977433 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:12.977448 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:12.978945 (+ 1497us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:12.979239 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:12.979242 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:12.979242 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:12.979299 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:12.979443 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:12.979444 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:12.979445 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:12.979616 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:12.979727 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:12.979846 (+ 119us) server_negotiation.cc:300] Negotiation successful
0319 16:33:12.979963 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260319 16:33:12.980942 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:33:12.996799 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:12.994101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.24.187.254:37055)
0319 16:33:12.994151 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:12.994153 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:12.994266 (+ 113us) server_negotiation.cc:408] Connection header received
0319 16:33:12.994332 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:12.994333 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:12.994345 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:12.994394 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:12.994874 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:12.995326 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:12.995828 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:12.996253 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:12.996355 (+ 102us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:12.996358 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:12.996359 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:12.996558 (+ 199us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:12.996573 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:12.996578 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:12.996627 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:12.996693 (+ 66us) server_negotiation.cc:300] Negotiation successful
0319 16:33:12.996719 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:33:13.005101 28906 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:13.005774 28906 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:13.016203 28906 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9773d73805c940cab4788dd01eea2750: Generated new TSK 0
I20260319 16:33:13.016386 28906 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:33:13.023075 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 28829
2026-03-19T16:33:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2132 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:13 dist-test-slave-svn8 krb5kdc[28930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:13 dist-test-slave-svn8 krb5kdc[28930](info): set up 2 sockets
Mar 19 16:33:13 dist-test-slave-svn8 krb5kdc[28930](info): commencing operation
krb5kdc: starting...
W20260319 16:33:14.870379 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.830s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:14 dist-test-slave-svn8 krb5kdc[28930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:15.004606 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42853
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:15.078858 28946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:15.079047 28946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:15.079078 28946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:15.080611 28946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:15.080657 28946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:15.080674 28946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:15.080690 28946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:15.080705 28946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:15.082407 28946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:42853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:15.082650 28946 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:15.082828 28946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:33:15.085134 28946 server_base.cc:1060] running on GCE node
W20260319 16:33:15.085105 28952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:15.085237 28951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:15.085112 28954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:15.085546 28946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:15.085824 28946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:15.086956 28946 hybrid_clock.cc:648] HybridClock initialized: now 1773937995086939 us; error 26 us; skew 500 ppm
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:15.089200 28946 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:15.089742 28946 webserver.cc:492] Webserver started at http://127.24.187.254:33129/ using document root <none> and password file <none>
I20260319 16:33:15.089902 28946 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:15.089934 28946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:15.090013 28946 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:15.090868 28946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "07fff52540334ff99c3e54a582fef1f8"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "4d19b0c3d2f9967ebc9b594161c67bf0"
server_key_iv: "c8b9d785a389e1350c706bd9fb3dd87f"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.091125 28946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "07fff52540334ff99c3e54a582fef1f8"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "4d19b0c3d2f9967ebc9b594161c67bf0"
server_key_iv: "c8b9d785a389e1350c706bd9fb3dd87f"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.092175 28946 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:33:15.092805 28961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.092975 28946 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.093037 28946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "07fff52540334ff99c3e54a582fef1f8"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "4d19b0c3d2f9967ebc9b594161c67bf0"
server_key_iv: "c8b9d785a389e1350c706bd9fb3dd87f"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.093089 28946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:15.122498 28946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:15.124679 28946 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:15.124789 28946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:15.128798 28946 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:42853
I20260319 16:33:15.128849 29013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:42853 every 8 connection(s)
I20260319 16:33:15.129114 28946 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:15.129278 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 28946
I20260319 16:33:15.129383 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:15.129568 25327 external_mini_cluster.cc:1468] Setting key 67339ae9f8d3bc5496b1736b4bec51da
I20260319 16:33:15.129763 29014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:15.133015 29014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8: Bootstrap starting.
I20260319 16:33:15.133687 29014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:15.133982 29014 log.cc:826] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:15.134757 29014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8: No bootstrap required, opened a new log
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:15.135833 29014 raft_consensus.cc:359] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fff52540334ff99c3e54a582fef1f8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42853 } }
I20260319 16:33:15.135953 29014 raft_consensus.cc:385] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:15.135982 29014 raft_consensus.cc:740] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fff52540334ff99c3e54a582fef1f8, State: Initialized, Role: FOLLOWER
I20260319 16:33:15.136077 29014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fff52540334ff99c3e54a582fef1f8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42853 } }
I20260319 16:33:15.136143 29014 raft_consensus.cc:399] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:15.136173 29014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:15.136219 29014 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:15.136713 29014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fff52540334ff99c3e54a582fef1f8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42853 } }
I20260319 16:33:15.136812 29014 leader_election.cc:304] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07fff52540334ff99c3e54a582fef1f8; no voters:
I20260319 16:33:15.136958 29014 leader_election.cc:290] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:15.137004 29019 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:15.137118 29019 raft_consensus.cc:697] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [term 1 LEADER]: Becoming Leader. State: Replica: 07fff52540334ff99c3e54a582fef1f8, State: Running, Role: LEADER
I20260319 16:33:15.137181 29014 sys_catalog.cc:565] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:15.137264 29019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fff52540334ff99c3e54a582fef1f8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42853 } }
I20260319 16:33:15.137580 29019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07fff52540334ff99c3e54a582fef1f8. Latest consensus state: current_term: 1 leader_uuid: "07fff52540334ff99c3e54a582fef1f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fff52540334ff99c3e54a582fef1f8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42853 } } }
I20260319 16:33:15.137652 29019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:15.138028 29021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07fff52540334ff99c3e54a582fef1f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fff52540334ff99c3e54a582fef1f8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42853 } } }
I20260319 16:33:15.138108 29021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:15.138443 29025 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:15.138917 29025 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:15.139536 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.130103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.24.187.254:42853)
0319 16:33:15.130256 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:15.130265 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:15.130283 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:33:15.130765 (+ 482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:15.130770 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:15.130807 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:15.130902 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:15.131865 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.132591 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.133367 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.133635 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.135745 (+ 2110us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:15.135752 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:15.135752 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:15.135769 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:15.137648 (+ 1879us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.138057 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.138060 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.138060 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.138120 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.138296 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.138297 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.138297 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.138547 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:15.139063 (+ 516us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:15.139148 (+ 85us) server_negotiation.cc:300] Negotiation successful
0319 16:33:15.139263 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260319 16:33:15.140578 29025 catalog_manager.cc:1357] Generated new cluster ID: e947bc663fe94feaad8747df21f6eac2
I20260319 16:33:15.140635 29025 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:15.161213 29025 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:15.161942 29025 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:15.173913 29025 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 07fff52540334ff99c3e54a582fef1f8: Generated new TSK 0
I20260319 16:33:15.174139 29025 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:15.237159 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42853
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:15.345436 29042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:15.345630 29042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:15.345664 29042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:15.347208 29042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:15.347256 29042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:15.347288 29042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:15.349030 29042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:15.349243 29042 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:15.349514 29042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:15.351984 29047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:15.351981 29050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:15.352123 29042 server_base.cc:1060] running on GCE node
W20260319 16:33:15.352056 29048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:15.352391 29042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:15.352566 29042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:15.353754 29042 hybrid_clock.cc:648] HybridClock initialized: now 1773937995353734 us; error 31 us; skew 500 ppm
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:15.356103 29042 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:15.356637 29042 webserver.cc:492] Webserver started at http://127.24.187.193:44979/ using document root <none> and password file <none>
I20260319 16:33:15.356805 29042 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:15.356840 29042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:15.356940 29042 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:15.357879 29042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "f0c5ece79b074991953565cba787a776"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "3419f40ad9f8eaccc7adcad90bda125e"
server_key_iv: "14fc853775c021a891fbc757a9894892"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.358175 29042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "f0c5ece79b074991953565cba787a776"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "3419f40ad9f8eaccc7adcad90bda125e"
server_key_iv: "14fc853775c021a891fbc757a9894892"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.359308 29042 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:15.359968 29057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.360131 29042 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:15.360188 29042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "f0c5ece79b074991953565cba787a776"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "3419f40ad9f8eaccc7adcad90bda125e"
server_key_iv: "14fc853775c021a891fbc757a9894892"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.360245 29042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:15.398389 29042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:15.400166 29042 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:15.400290 29042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:15.400499 29042 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:15.400826 29042 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:15.400905 29042 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.400974 29042 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:15.401006 29042 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.406859 29042 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:39879
I20260319 16:33:15.406919 29170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:39879 every 8 connection(s)
I20260319 16:33:15.407184 29042 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:15.413892 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29042
I20260319 16:33:15.414012 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:15.414183 25327 external_mini_cluster.cc:1468] Setting key 1e33de20f3d2c0e6ed87e0f321f03874
I20260319 16:33:15.415589 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.407892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:57303 (local address 127.24.187.254:42853)
0319 16:33:15.407935 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:15.407937 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:15.408078 (+ 141us) server_negotiation.cc:408] Connection header received
0319 16:33:15.408561 (+ 483us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:15.408562 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:15.408571 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:15.408631 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:15.409406 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.409953 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.410568 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.410693 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.412913 (+ 2220us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:15.412916 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:15.412917 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:15.412924 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:15.414407 (+ 1483us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.414782 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.414784 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.414784 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.414814 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.415027 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.415028 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.415029 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.415113 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:15.415170 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:15.415406 (+ 236us) server_negotiation.cc:300] Negotiation successful
0319 16:33:15.415500 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:33:15.415691 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.407854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42853 (local address 127.24.187.193:57303)
0319 16:33:15.407990 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:15.408020 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:15.408489 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:15.408726 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:15.408730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:15.409005 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:15.409279 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:15.409282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.410051 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.410052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:15.410486 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:15.410488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.410603 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.411028 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:15.411043 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:15.412831 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:15.414484 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:15.414490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:15.414492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:15.414703 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:15.414901 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:15.414902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:15.414903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:15.414965 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:15.415173 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:15.415178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:15.415293 (+ 115us) client_negotiation.cc:769] Sending connection context
0319 16:33:15.415380 (+ 87us) client_negotiation.cc:240] Negotiation successful
0319 16:33:15.415509 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260319 16:33:15.416399 29171 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42853
I20260319 16:33:15.416482 29171 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:15.416661 29171 heartbeater.cc:507] Master 127.24.187.254:42853 requested a full tablet report, sending...
I20260319 16:33:15.417167 28978 ts_manager.cc:194] Registered new tserver with Master: f0c5ece79b074991953565cba787a776 (127.24.187.193:39879)
I20260319 16:33:15.417893 28978 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:57303
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:15.460371 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42853
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:15.532886 29178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:15.533046 29178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:15.533066 29178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:15.534534 29178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:15.534575 29178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:15.534600 29178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:15.536167 29178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:15.536339 29178 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:15.536501 29178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:15.539166 29183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:15.539179 29184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:15.539230 29178 server_base.cc:1060] running on GCE node
W20260319 16:33:15.539175 29186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:15.539598 29178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:15.539808 29178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:15.540942 29178 hybrid_clock.cc:648] HybridClock initialized: now 1773937995540922 us; error 31 us; skew 500 ppm
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:15.543730 29178 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:15.544360 29178 webserver.cc:492] Webserver started at http://127.24.187.194:37125/ using document root <none> and password file <none>
I20260319 16:33:15.544570 29178 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:15.544617 29178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:15.544742 29178 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:15.545746 29178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "b1cc9085f30440c8a344f52bcedd7739"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "35b045b4cbbffe8540bc363e14dd245d"
server_key_iv: "80483421cdd6c2ae0d4146bdd26b190d"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.546018 29178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "b1cc9085f30440c8a344f52bcedd7739"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "35b045b4cbbffe8540bc363e14dd245d"
server_key_iv: "80483421cdd6c2ae0d4146bdd26b190d"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.547124 29178 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:15.547776 29193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.547937 29178 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:15.547998 29178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "b1cc9085f30440c8a344f52bcedd7739"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "35b045b4cbbffe8540bc363e14dd245d"
server_key_iv: "80483421cdd6c2ae0d4146bdd26b190d"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.548043 29178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:15.583726 29178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:15.585340 29178 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:15.585491 29178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:15.585676 29178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:15.586005 29178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:15.586038 29178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.586061 29178 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:15.586071 29178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.591480 29178 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:46069
I20260319 16:33:15.591538 29306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:46069 every 8 connection(s)
I20260319 16:33:15.591794 29178 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:15.594923 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29178
I20260319 16:33:15.595021 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:15.595193 25327 external_mini_cluster.cc:1468] Setting key 1f9a6f9ee195d4af6a961c143ef70e77
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:15.602276 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.592419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:50755 (local address 127.24.187.254:42853)
0319 16:33:15.592471 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:15.592473 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:15.592594 (+ 121us) server_negotiation.cc:408] Connection header received
0319 16:33:15.593080 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:15.593082 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:15.593092 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:15.593144 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:15.593949 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.594446 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.595048 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.595187 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.597567 (+ 2380us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:15.597570 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:15.597571 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:15.597579 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:15.598852 (+ 1273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.599274 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.599275 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.599275 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.599295 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.599481 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.599482 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.599482 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.599566 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:15.599614 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:15.602083 (+ 2469us) server_negotiation.cc:300] Negotiation successful
0319 16:33:15.602183 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:33:15.602430 29309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.592354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42853 (local address 127.24.187.194:50755)
0319 16:33:15.592493 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:15.592520 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:15.593002 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:15.593233 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:15.593237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:15.593526 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:15.593867 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:15.593871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.594517 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.594518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:15.594952 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:15.594954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.595075 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.595549 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:15.595564 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:15.597486 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:15.598938 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:15.598944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:15.598946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:15.599206 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:15.599348 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:15.599349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:15.599349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:15.599427 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:15.601816 (+ 2389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:15.601821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:15.601938 (+ 117us) client_negotiation.cc:769] Sending connection context
0319 16:33:15.602046 (+ 108us) client_negotiation.cc:240] Negotiation successful
0319 16:33:15.602188 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
I20260319 16:33:15.603011 29307 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42853
I20260319 16:33:15.603111 29307 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:15.603299 29307 heartbeater.cc:507] Master 127.24.187.254:42853 requested a full tablet report, sending...
I20260319 16:33:15.603705 28978 ts_manager.cc:194] Registered new tserver with Master: b1cc9085f30440c8a344f52bcedd7739 (127.24.187.194:46069)
I20260319 16:33:15.604139 28978 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:50755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:15.640730 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42853
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:15.714996 29314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:15.715168 29314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:15.715189 29314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:15.716603 29314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:15.716642 29314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:15.716666 29314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:15.718248 29314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:15.718421 29314 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:15.718550 29314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:15.720870 29320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:15.720858 29319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:15.720858 29322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:15.721005 29314 server_base.cc:1060] running on GCE node
I20260319 16:33:15.721202 29314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:15.721432 29314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:15.722585 29314 hybrid_clock.cc:648] HybridClock initialized: now 1773937995722567 us; error 27 us; skew 500 ppm
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:15.724845 29314 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:15.725292 29314 webserver.cc:492] Webserver started at http://127.24.187.195:45895/ using document root <none> and password file <none>
I20260319 16:33:15.725450 29314 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:15.725483 29314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:15.725567 29314 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:15.726500 29314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "60266863ccb34d9fb8b3b312c3800814"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "6bf8b4e5b5edf5e3cb82ad6f0910f2f9"
server_key_iv: "bc0e92dd0fc8a01525b6dc6cd8ff34fb"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.726770 29314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "60266863ccb34d9fb8b3b312c3800814"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "6bf8b4e5b5edf5e3cb82ad6f0910f2f9"
server_key_iv: "bc0e92dd0fc8a01525b6dc6cd8ff34fb"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.727839 29314 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:15.728549 29329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.728742 29314 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:15.728804 29314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "60266863ccb34d9fb8b3b312c3800814"
format_stamp: "Formatted at 2026-03-19 16:33:15 on dist-test-slave-svn8"
server_key: "6bf8b4e5b5edf5e3cb82ad6f0910f2f9"
server_key_iv: "bc0e92dd0fc8a01525b6dc6cd8ff34fb"
server_key_version: "encryptionkey@0"
I20260319 16:33:15.728855 29314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:15.748934 29314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:15.750674 29314 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:15.750770 29314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:15.750933 29314 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:15.751225 29314 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:15.751256 29314 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.751276 29314 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:15.751287 29314 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:15.756453 29314 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:34537
I20260319 16:33:15.756533 29442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:34537 every 8 connection(s)
I20260319 16:33:15.756754 29314 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:15.764222 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.757342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:33705 (local address 127.24.187.254:42853)
0319 16:33:15.757397 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:15.757399 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:15.757560 (+ 161us) server_negotiation.cc:408] Connection header received
0319 16:33:15.758081 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:15.758083 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:15.758091 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:15.758131 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:15.758776 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.759242 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.759780 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.759926 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.762069 (+ 2143us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:15.762073 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:15.762073 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:15.762079 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:15.763188 (+ 1109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.763545 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.763546 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.763547 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.763566 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.763748 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.763748 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.763748 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.763846 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:15.763894 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:15.764091 (+ 197us) server_negotiation.cc:300] Negotiation successful
0319 16:33:15.764160 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:33:15.764317 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.757299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42853 (local address 127.24.187.195:33705)
0319 16:33:15.757464 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:15.757494 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:15.757989 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:15.758173 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:15.758177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:15.758448 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:15.758713 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:15.758716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.759303 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.759303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:15.759674 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:15.759675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.759762 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.760165 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:15.760178 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:15.761992 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:15.763262 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:15.763266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:15.763268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:15.763488 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:15.763635 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:15.763636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:15.763636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:15.763699 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:15.763895 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:15.763899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:15.763994 (+ 95us) client_negotiation.cc:769] Sending connection context
0319 16:33:15.764075 (+ 81us) client_negotiation.cc:240] Negotiation successful
0319 16:33:15.764172 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260319 16:33:15.764784 29443 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42853
I20260319 16:33:15.764856 29443 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:15.765019 29443 heartbeater.cc:507] Master 127.24.187.254:42853 requested a full tablet report, sending...
I20260319 16:33:15.765065 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29314
I20260319 16:33:15.765156 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:15.765326 25327 external_mini_cluster.cc:1468] Setting key 41d29ecf9fc7dfc9e1a88745233ad8d3
I20260319 16:33:15.765378 28978 ts_manager.cc:194] Registered new tserver with Master: 60266863ccb34d9fb8b3b312c3800814 (127.24.187.195:34537)
I20260319 16:33:15.765820 28978 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:33705
I20260319 16:33:15.766105 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[28930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937995, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:15.786069 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:15.780246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.24.187.254:42853)
0319 16:33:15.780312 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:15.780315 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:15.780376 (+ 61us) server_negotiation.cc:408] Connection header received
0319 16:33:15.780445 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:15.780446 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:15.780458 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:15.780506 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:15.781147 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.781617 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:15.782171 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:15.782330 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:15.784096 (+ 1766us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:15.784100 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:15.784100 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:15.784107 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:15.785105 (+ 998us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.785362 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.785363 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.785363 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.785384 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:15.785543 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:15.785543 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:15.785544 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:15.785652 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:15.785819 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:15.785898 (+ 79us) server_negotiation.cc:300] Negotiation successful
0319 16:33:15.785977 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260319 16:33:15.786290 28977 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45856
I20260319 16:33:15.786660 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29042
I20260319 16:33:15.791501 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29178
I20260319 16:33:15.795429 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29314
I20260319 16:33:15.799322 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 28946
2026-03-19T16:33:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2777 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[29455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[29455](info): set up 2 sockets
Mar 19 16:33:15 dist-test-slave-svn8 krb5kdc[29455](info): commencing operation
krb5kdc: starting...
W20260319 16:33:17.634473 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:17 dist-test-slave-svn8 krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:17.766289 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:44945
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:17.841126 29471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:17.841302 29471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:17.841332 29471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:17.842871 29471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:17.842922 29471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:17.842942 29471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:17.842957 29471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:17.842972 29471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:17.845098 29471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:17.845350 29471 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:17.845528 29471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:17.847991 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:17.847997 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:17.848023 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:17.848256 29471 server_base.cc:1060] running on GCE node
I20260319 16:33:17.848537 29471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:17.848785 29471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:17.849913 29471 hybrid_clock.cc:648] HybridClock initialized: now 1773937997849898 us; error 25 us; skew 500 ppm
Mar 19 16:33:17 dist-test-slave-svn8 krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937997, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:17.852061 29471 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:17.852567 29471 webserver.cc:492] Webserver started at http://127.24.187.254:33305/ using document root <none> and password file <none>
I20260319 16:33:17.852751 29471 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:17.852787 29471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:17.852883 29471 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:17.853843 29471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "94e6cd1f6e9946b584ca933005ec0c35"
format_stamp: "Formatted at 2026-03-19 16:33:17 on dist-test-slave-svn8"
server_key: "a718c55f7e8261b4073f467f9c7cfff4"
server_key_iv: "ec74f855d35373833815c8bcaf08206d"
server_key_version: "encryptionkey@0"
I20260319 16:33:17.854127 29471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "94e6cd1f6e9946b584ca933005ec0c35"
format_stamp: "Formatted at 2026-03-19 16:33:17 on dist-test-slave-svn8"
server_key: "a718c55f7e8261b4073f467f9c7cfff4"
server_key_iv: "ec74f855d35373833815c8bcaf08206d"
server_key_version: "encryptionkey@0"
I20260319 16:33:17.855243 29471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:17.856003 29486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:17.856222 29471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:17.856283 29471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "94e6cd1f6e9946b584ca933005ec0c35"
format_stamp: "Formatted at 2026-03-19 16:33:17 on dist-test-slave-svn8"
server_key: "a718c55f7e8261b4073f467f9c7cfff4"
server_key_iv: "ec74f855d35373833815c8bcaf08206d"
server_key_version: "encryptionkey@0"
I20260319 16:33:17.856343 29471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:17.872910 29471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:17.874871 29471 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:17.874986 29471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:17.878621 29471 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:44945
I20260319 16:33:17.878661 29538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:44945 every 8 connection(s)
I20260319 16:33:17.878930 29471 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:17.879487 29539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:17.880515 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29471
I20260319 16:33:17.880609 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:17.880759 25327 external_mini_cluster.cc:1468] Setting key 8d32ef7554a84b9e2d156c55b656d5de
I20260319 16:33:17.882519 29539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35: Bootstrap starting.
I20260319 16:33:17.883091 29539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:17.883318 29539 log.cc:826] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:17.883901 29539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35: No bootstrap required, opened a new log
I20260319 16:33:17.885043 29539 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e6cd1f6e9946b584ca933005ec0c35" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 44945 } }
I20260319 16:33:17.885157 29539 raft_consensus.cc:385] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:17.885185 29539 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e6cd1f6e9946b584ca933005ec0c35, State: Initialized, Role: FOLLOWER
I20260319 16:33:17.885268 29539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e6cd1f6e9946b584ca933005ec0c35" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 44945 } }
I20260319 16:33:17.885339 29539 raft_consensus.cc:399] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:17.885370 29539 raft_consensus.cc:493] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:17.885412 29539 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 0 FOLLOWER]: Advancing to term 1
Mar 19 16:33:17 dist-test-slave-svn8 krb5kdc[29455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:17.885989 29539 raft_consensus.cc:515] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e6cd1f6e9946b584ca933005ec0c35" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 44945 } }
I20260319 16:33:17.886094 29539 leader_election.cc:304] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94e6cd1f6e9946b584ca933005ec0c35; no voters:
I20260319 16:33:17.886224 29539 leader_election.cc:290] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:17.886281 29544 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:17.886387 29544 raft_consensus.cc:697] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [term 1 LEADER]: Becoming Leader. State: Replica: 94e6cd1f6e9946b584ca933005ec0c35, State: Running, Role: LEADER
I20260319 16:33:17.886453 29539 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:17.886461 29544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e6cd1f6e9946b584ca933005ec0c35" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 44945 } }
I20260319 16:33:17.886821 29546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94e6cd1f6e9946b584ca933005ec0c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e6cd1f6e9946b584ca933005ec0c35" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 44945 } } }
I20260319 16:33:17.886898 29546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:17.887015 29546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e6cd1f6e9946b584ca933005ec0c35. Latest consensus state: current_term: 1 leader_uuid: "94e6cd1f6e9946b584ca933005ec0c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e6cd1f6e9946b584ca933005ec0c35" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 44945 } } }
I20260319 16:33:17.887075 29546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:17.887198 29550 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:17.887640 29550 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:17.889696 29561 mvcc.cc:204] Tried to move back new op lower bound from 7266050039350882304 to 7266050039343542272. Current Snapshot: MvccSnapshot[applied={T|T < 7266050039350882304}]
I20260319 16:33:17.890112 29550 catalog_manager.cc:1357] Generated new cluster ID: 32c2110d233f42659505186dd6ccc7e0
I20260319 16:33:17.890163 29550 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:17.890681 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:17.881213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.24.187.254:44945)
0319 16:33:17.881335 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:17.881343 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:17.881360 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:33:17.881691 (+ 331us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:17.881697 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:17.881769 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:17.881865 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:17.882767 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:17.883525 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:17.884061 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:17.884322 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:17.886429 (+ 2107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:17.886438 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:17.886439 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:17.886459 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:17.889133 (+ 2674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:17.889441 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:17.889445 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:17.889445 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:17.889506 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:17.889700 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:17.889701 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:17.889701 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:17.889963 (+ 262us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:17.890116 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:17.890366 (+ 250us) server_negotiation.cc:300] Negotiation successful
0319 16:33:17.890489 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260319 16:33:17.901320 29550 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:17.901913 29550 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:17.914785 29550 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 94e6cd1f6e9946b584ca933005ec0c35: Generated new TSK 0
I20260319 16:33:17.914953 29550 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:17.964859 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:44945
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:18.038424 29567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:18.038580 29567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:18.038601 29567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:18.040019 29567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:18.040066 29567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:18.040092 29567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:18.041682 29567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:18.041909 29567 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:18.042064 29567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:18.044555 29572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:18.044721 29573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:18.044526 29575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:18.044943 29567 server_base.cc:1060] running on GCE node
I20260319 16:33:18.045118 29567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:18.045303 29567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:18.046502 29567 hybrid_clock.cc:648] HybridClock initialized: now 1773937998046485 us; error 32 us; skew 500 ppm
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:18.048754 29567 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:18.049245 29567 webserver.cc:492] Webserver started at http://127.24.187.193:41499/ using document root <none> and password file <none>
I20260319 16:33:18.049422 29567 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:18.049456 29567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:18.049543 29567 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:18.050494 29567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "94cf47666104464fb1e994966965e411"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "67c8958797bde0953aced16f61fe6bc8"
server_key_iv: "3a72add49d3430d04df4dd6e7cd3cb29"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.050762 29567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "94cf47666104464fb1e994966965e411"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "67c8958797bde0953aced16f61fe6bc8"
server_key_iv: "3a72add49d3430d04df4dd6e7cd3cb29"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.051843 29567 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:18.052564 29582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.052768 29567 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:18.052846 29567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "94cf47666104464fb1e994966965e411"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "67c8958797bde0953aced16f61fe6bc8"
server_key_iv: "3a72add49d3430d04df4dd6e7cd3cb29"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.052894 29567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:18.060503 29567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:18.066937 29567 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:18.067041 29567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:18.067188 29567 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:18.067483 29567 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:18.067520 29567 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.067541 29567 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:18.067559 29567 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.072860 29567 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:34993
I20260319 16:33:18.072914 29695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:34993 every 8 connection(s)
I20260319 16:33:18.073169 29567 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:18.079421 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29567
I20260319 16:33:18.079505 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:18.079656 25327 external_mini_cluster.cc:1468] Setting key 4de2bfadbd97cabf10e4fb454bd441e2
I20260319 16:33:18.081132 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.073913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:55943 (local address 127.24.187.254:44945)
0319 16:33:18.073976 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:18.073978 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:18.074115 (+ 137us) server_negotiation.cc:408] Connection header received
0319 16:33:18.074595 (+ 480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:18.074596 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:18.074606 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:18.074668 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:18.075333 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.075827 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.076373 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.076497 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.078625 (+ 2128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:18.078630 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:18.078631 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:18.078638 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:18.079992 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.080415 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.080416 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.080417 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.080444 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.080600 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.080600 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.080601 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.080715 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:18.080772 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:18.080965 (+ 193us) server_negotiation.cc:300] Negotiation successful
0319 16:33:18.081054 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260319 16:33:18.081158 29698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.073874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:44945 (local address 127.24.187.193:55943)
0319 16:33:18.074013 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:18.074042 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:18.074499 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:18.074716 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:18.074719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:18.074993 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:18.075264 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:18.075267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.075921 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.075922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:18.076308 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:18.076310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.076400 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.077190 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:18.077202 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:18.078530 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:18.080072 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:18.080078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:18.080080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:18.080331 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:18.080506 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:18.080507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:18.080507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:18.080554 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:18.080769 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:18.080771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:18.080863 (+ 92us) client_negotiation.cc:769] Sending connection context
0319 16:33:18.080935 (+ 72us) client_negotiation.cc:240] Negotiation successful
0319 16:33:18.081031 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260319 16:33:18.081741 29696 heartbeater.cc:344] Connected to a master server at 127.24.187.254:44945
I20260319 16:33:18.081821 29696 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:18.081987 29696 heartbeater.cc:507] Master 127.24.187.254:44945 requested a full tablet report, sending...
I20260319 16:33:18.082471 29503 ts_manager.cc:194] Registered new tserver with Master: 94cf47666104464fb1e994966965e411 (127.24.187.193:34993)
I20260319 16:33:18.083070 29503 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:55943
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:18.125481 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:44945
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:18.200132 29703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:18.200284 29703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:18.200305 29703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:18.201702 29703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:18.201771 29703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:18.201794 29703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:18.203532 29703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:18.203707 29703 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:18.203845 29703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:18.206306 29708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:18.206307 29709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:18.206355 29703 server_base.cc:1060] running on GCE node
W20260319 16:33:18.206307 29711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:18.206732 29703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:18.206902 29703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:18.208048 29703 hybrid_clock.cc:648] HybridClock initialized: now 1773937998208032 us; error 30 us; skew 500 ppm
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:18.210403 29703 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:18.210894 29703 webserver.cc:492] Webserver started at http://127.24.187.194:40829/ using document root <none> and password file <none>
I20260319 16:33:18.211035 29703 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:18.211063 29703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:18.211143 29703 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:18.212011 29703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "83045a53c51f4326bf8123225dd65bdc"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "ce235de0353305817d8d1901eec135d3"
server_key_iv: "a7f37bf0cc90530ab5f9255841d577ba"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.212258 29703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "83045a53c51f4326bf8123225dd65bdc"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "ce235de0353305817d8d1901eec135d3"
server_key_iv: "a7f37bf0cc90530ab5f9255841d577ba"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.213327 29703 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:18.214069 29718 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.214254 29703 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:18.214313 29703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "83045a53c51f4326bf8123225dd65bdc"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "ce235de0353305817d8d1901eec135d3"
server_key_iv: "a7f37bf0cc90530ab5f9255841d577ba"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.214361 29703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:18.222705 29703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:18.224419 29703 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:18.224514 29703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:18.224666 29703 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:18.224908 29703 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:18.224936 29703 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.224956 29703 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:18.224967 29703 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.230125 29703 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:38193
I20260319 16:33:18.230182 29831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:38193 every 8 connection(s)
I20260319 16:33:18.230420 29703 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:18.238006 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.231009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:44283 (local address 127.24.187.254:44945)
0319 16:33:18.231058 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:18.231061 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:18.231187 (+ 126us) server_negotiation.cc:408] Connection header received
0319 16:33:18.231681 (+ 494us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:18.231683 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:18.231695 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:18.231744 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:18.232437 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.232928 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.233450 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.233619 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.235788 (+ 2169us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:18.235793 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:18.235793 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:18.235800 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:18.237029 (+ 1229us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.237352 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.237353 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.237353 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.237370 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.237501 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.237501 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.237502 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.237588 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:18.237636 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:18.237872 (+ 236us) server_negotiation.cc:300] Negotiation successful
0319 16:33:18.237942 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:33:18.238080 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.230971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:44945 (local address 127.24.187.194:44283)
0319 16:33:18.231085 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:18.231115 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:18.231588 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:18.231794 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:18.231797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:18.232060 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:18.232317 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:18.232320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.232998 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.232999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:18.233380 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:18.233382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.233482 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.234384 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:18.234397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:18.235706 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:18.237110 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:18.237115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:18.237117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:18.237303 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:18.237413 (+ 110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:18.237414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:18.237415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:18.237460 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:18.237629 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:18.237632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:18.237763 (+ 131us) client_negotiation.cc:769] Sending connection context
0319 16:33:18.237844 (+ 81us) client_negotiation.cc:240] Negotiation successful
0319 16:33:18.237941 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":54,"threads_started":1}
I20260319 16:33:18.238624 29832 heartbeater.cc:344] Connected to a master server at 127.24.187.254:44945
I20260319 16:33:18.238706 29832 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:18.238855 29832 heartbeater.cc:507] Master 127.24.187.254:44945 requested a full tablet report, sending...
I20260319 16:33:18.239166 29503 ts_manager.cc:194] Registered new tserver with Master: 83045a53c51f4326bf8123225dd65bdc (127.24.187.194:38193)
I20260319 16:33:18.239531 29503 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:44283
I20260319 16:33:18.240003 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29703
I20260319 16:33:18.240080 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:18.240239 25327 external_mini_cluster.cc:1468] Setting key e40977ca1f192fab57a7332bc4eb1ff9
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:18.283629 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:44945
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:18.357228 29839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:18.357390 29839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:18.357411 29839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:18.358887 29839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:18.358937 29839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:18.358963 29839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:18.360591 29839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:18.360796 29839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:18.360951 29839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:18.363361 29847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:18.363411 29845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:18.363372 29844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:18.363543 29839 server_base.cc:1060] running on GCE node
I20260319 16:33:18.363698 29839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:18.363862 29839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:18.364985 29839 hybrid_clock.cc:648] HybridClock initialized: now 1773937998364971 us; error 28 us; skew 500 ppm
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:18.367192 29839 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:18.367657 29839 webserver.cc:492] Webserver started at http://127.24.187.195:37035/ using document root <none> and password file <none>
I20260319 16:33:18.367807 29839 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:18.367836 29839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:18.367906 29839 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:18.368775 29839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "1a8ed8ace9c948bab6f97c44a631e9ee"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "7f445f02f963e6d81da16c6dda02b9f8"
server_key_iv: "dc1b5638f532f58cdf5d61f8bfef3b6d"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.369021 29839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "1a8ed8ace9c948bab6f97c44a631e9ee"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "7f445f02f963e6d81da16c6dda02b9f8"
server_key_iv: "dc1b5638f532f58cdf5d61f8bfef3b6d"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.370057 29839 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260319 16:33:18.370676 29854 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.370862 29839 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.370923 29839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "1a8ed8ace9c948bab6f97c44a631e9ee"
format_stamp: "Formatted at 2026-03-19 16:33:18 on dist-test-slave-svn8"
server_key: "7f445f02f963e6d81da16c6dda02b9f8"
server_key_iv: "dc1b5638f532f58cdf5d61f8bfef3b6d"
server_key_version: "encryptionkey@0"
I20260319 16:33:18.370985 29839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:18.388622 29839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:18.390201 29839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:18.390295 29839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:18.390446 29839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:18.390754 29839 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:18.390795 29839 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.390831 29839 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:18.390853 29839 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:18.396121 29839 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:36217
I20260319 16:33:18.396179 29967 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:36217 every 8 connection(s)
I20260319 16:33:18.396413 29839 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:33:18.398068 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29839
I20260319 16:33:18.398135 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:18.398265 25327 external_mini_cluster.cc:1468] Setting key 556e7528d349ccf2378b4647f02893d2
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:18.404556 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.397049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:53719 (local address 127.24.187.254:44945)
0319 16:33:18.397124 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:18.397126 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:18.397312 (+ 186us) server_negotiation.cc:408] Connection header received
0319 16:33:18.397804 (+ 492us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:18.397806 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:18.397815 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:18.397865 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:18.398720 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.399328 (+ 608us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.399866 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.399969 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.402227 (+ 2258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:18.402232 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:18.402232 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:18.402240 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:18.403490 (+ 1250us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.403831 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.403832 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.403832 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.403853 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.404004 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.404005 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.404006 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.404093 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:18.404145 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:18.404370 (+ 225us) server_negotiation.cc:300] Negotiation successful
0319 16:33:18.404463 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260319 16:33:18.404640 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.396994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:44945 (local address 127.24.187.195:53719)
0319 16:33:18.397188 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:18.397215 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:18.397698 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:18.397944 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:18.397948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:18.398265 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:18.398628 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:18.398633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.399415 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.399416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:18.399804 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:18.399806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.399898 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.400722 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:18.400736 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:18.402118 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:18.403564 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:18.403569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:18.403572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:18.403769 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:18.403902 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:18.403902 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:18.403903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:18.403957 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:18.404142 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:18.404146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:18.404257 (+ 111us) client_negotiation.cc:769] Sending connection context
0319 16:33:18.404347 (+ 90us) client_negotiation.cc:240] Negotiation successful
0319 16:33:18.404469 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
I20260319 16:33:18.405087 29968 heartbeater.cc:344] Connected to a master server at 127.24.187.254:44945
I20260319 16:33:18.405169 29968 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:18.405350 29968 heartbeater.cc:507] Master 127.24.187.254:44945 requested a full tablet report, sending...
I20260319 16:33:18.405733 29503 ts_manager.cc:194] Registered new tserver with Master: 1a8ed8ace9c948bab6f97c44a631e9ee (127.24.187.195:36217)
I20260319 16:33:18.406095 29503 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:53719
I20260319 16:33:18.410135 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:33:18.431306 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.424406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.24.187.193:34993)
0319 16:33:18.424563 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:18.424569 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:18.424580 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:18.424608 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:18.424610 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:18.424623 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:18.424693 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:18.425394 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.426156 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.426722 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.426930 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.428801 (+ 1871us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:18.428830 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:18.428830 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:18.428837 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:18.430408 (+ 1571us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.430687 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.430690 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.430690 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.430719 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.430826 (+ 107us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.430826 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.430827 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.430934 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:18.431062 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:18.431109 (+ 47us) server_negotiation.cc:300] Negotiation successful
0319 16:33:18.431188 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773937998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:18.437943 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:18.432419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.24.187.254:44945)
0319 16:33:18.432490 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:18.432492 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:18.432501 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:18.432524 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:18.432525 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:18.432534 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:18.432576 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:18.433063 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.433547 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:18.434098 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:18.434208 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:18.436061 (+ 1853us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:18.436064 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:18.436064 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:18.436071 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:18.437111 (+ 1040us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.437366 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.437366 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.437367 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.437382 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:18.437496 (+ 114us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:18.437497 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:18.437497 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:18.437597 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:18.437672 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:18.437807 (+ 135us) server_negotiation.cc:300] Negotiation successful
0319 16:33:18.437866 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260319 16:33:18.438207 29503 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56666
I20260319 16:33:18.438519 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29567
I20260319 16:33:18.442796 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29703
I20260319 16:33:18.447039 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29839
I20260319 16:33:18.451007 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29471
2026-03-19T16:33:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2651 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29980](info): set up 2 sockets
Mar 19 16:33:18 dist-test-slave-svn8 krb5kdc[29980](info): commencing operation
krb5kdc: starting...
W20260319 16:33:20.290457 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.820s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:20 dist-test-slave-svn8 krb5kdc[29980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:20.432701 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38103
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:38103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:20.506541 29996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:20.506717 29996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:20.506748 29996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:20.508200 29996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:20.508247 29996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:20.508266 29996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:20.508281 29996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:20.508296 29996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:20.509977 29996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:38103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:20.510201 29996 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:20.510360 29996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:20.512848 30001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:20.512976 30002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:20.512882 29996 server_base.cc:1060] running on GCE node
W20260319 16:33:20.512856 30004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:20.513319 29996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:20.513566 29996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:20.514714 29996 hybrid_clock.cc:648] HybridClock initialized: now 1773938000514695 us; error 30 us; skew 500 ppm
Mar 19 16:33:20 dist-test-slave-svn8 krb5kdc[29980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938000, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:20.517023 29996 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:20.517325 29996 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:20.517369 29996 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:20.517474 29996 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:20.518504 29996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "1f2b243e495641a98e448cb4c2bc8c6b"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "8332ce13e04e7e8a3057051e2197ba69"
server_key_iv: "63098e2e86a6077d1c33083b182b91a9"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.518805 29996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "1f2b243e495641a98e448cb4c2bc8c6b"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "8332ce13e04e7e8a3057051e2197ba69"
server_key_iv: "63098e2e86a6077d1c33083b182b91a9"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.519881 29996 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:33:20.520527 30010 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.520720 29996 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:20.520778 29996 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "1f2b243e495641a98e448cb4c2bc8c6b"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "8332ce13e04e7e8a3057051e2197ba69"
server_key_iv: "63098e2e86a6077d1c33083b182b91a9"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.520841 29996 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:20.528805 29996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:20.530614 29996 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:20.530745 29996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:20.534409 29996 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:38103
I20260319 16:33:20.534443 30062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:38103 every 8 connection(s)
I20260319 16:33:20.534665 29996 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:20.535195 30063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:20.537102 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 29996
I20260319 16:33:20.537204 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:20.537350 25327 external_mini_cluster.cc:1468] Setting key a918e439ca6454a01a7d2f340bbd9043
I20260319 16:33:20.537575 30063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b: Bootstrap starting.
I20260319 16:33:20.538218 30063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:20.538503 30063 log.cc:826] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:20.539215 30063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b: No bootstrap required, opened a new log
I20260319 16:33:20.540294 30063 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38103 } }
I20260319 16:33:20.540409 30063 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:20.540428 30063 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f2b243e495641a98e448cb4c2bc8c6b, State: Initialized, Role: FOLLOWER
I20260319 16:33:20.540513 30063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38103 } }
I20260319 16:33:20.540570 30063 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:20.540599 30063 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:20.540642 30063 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:20.541282 30063 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38103 } }
I20260319 16:33:20.541385 30063 leader_election.cc:304] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f2b243e495641a98e448cb4c2bc8c6b; no voters:
I20260319 16:33:20.541530 30063 leader_election.cc:290] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:20.541594 30068 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:20.541783 30063 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:20.541791 30068 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [term 1 LEADER]: Becoming Leader. State: Replica: 1f2b243e495641a98e448cb4c2bc8c6b, State: Running, Role: LEADER
I20260319 16:33:20.541990 30068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38103 } }
I20260319 16:33:20.542470 30069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38103 } } }
I20260319 16:33:20.542640 30069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [sys.catalog]: This master's current role is: LEADER
Mar 19 16:33:20 dist-test-slave-svn8 krb5kdc[29980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:20.542984 30077 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:20.543246 30070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f2b243e495641a98e448cb4c2bc8c6b. Latest consensus state: current_term: 1 leader_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2b243e495641a98e448cb4c2bc8c6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38103 } } }
I20260319 16:33:20.543314 30070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:20.543497 30077 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:20.545220 30077 catalog_manager.cc:1357] Generated new cluster ID: 3b8d238384054b48b93631f02379f3e6
I20260319 16:33:20.545272 30077 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:20.546490 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:20.537877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.24.187.254:38103)
0319 16:33:20.538041 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:20.538052 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:20.538075 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:33:20.538444 (+ 369us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:20.538455 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:20.538507 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:20.538644 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:20.539472 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.540176 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:20.540841 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.541070 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:20.543488 (+ 2418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:20.543497 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:20.543497 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:20.543517 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:20.545317 (+ 1800us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:20.545596 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:20.545599 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:20.545599 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:20.545655 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:20.545827 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:20.545828 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:20.545828 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:20.546006 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:20.546143 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:20.546256 (+ 113us) server_negotiation.cc:300] Negotiation successful
0319 16:33:20.546371 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260319 16:33:20.558374 30077 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:20.559171 30077 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:20.568413 30077 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b: Generated new TSK 0
I20260319 16:33:20.568576 30077 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:20.618516 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38103
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:20.692873 30091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:20.693043 30091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:20.693076 30091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:20.694556 30091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:20.694607 30091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:20.694643 30091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:20.696241 30091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:20.696465 30091 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:20.696621 30091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:20.698884 30096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:20.698890 30099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:20.698894 30097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:20.699147 30091 server_base.cc:1060] running on GCE node
I20260319 16:33:20.699352 30091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:20.699553 30091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:20.700695 30091 hybrid_clock.cc:648] HybridClock initialized: now 1773938000700678 us; error 27 us; skew 500 ppm
Mar 19 16:33:20 dist-test-slave-svn8 krb5kdc[29980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938000, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:20.702939 30091 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:20.703199 30091 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:20.703236 30091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:20.703326 30091 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:20.704205 30091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "1b72891761c04ebc8b8473437a53d8b0"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "900a4567276c3da2d15a8ffe6e593541"
server_key_iv: "5ed371c454346a54ede285db46292a11"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.704483 30091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "1b72891761c04ebc8b8473437a53d8b0"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "900a4567276c3da2d15a8ffe6e593541"
server_key_iv: "5ed371c454346a54ede285db46292a11"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.705500 30091 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:20.706146 30105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.706316 30091 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.706374 30091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "1b72891761c04ebc8b8473437a53d8b0"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "900a4567276c3da2d15a8ffe6e593541"
server_key_iv: "5ed371c454346a54ede285db46292a11"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.706457 30091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:20.713922 30091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:20.715515 30091 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:20.715611 30091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:20.715775 30091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:20.716039 30091 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:20.716069 30091 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.716090 30091 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:20.716112 30091 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.721446 30091 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:38543
I20260319 16:33:20.721508 30218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:38543 every 8 connection(s)
I20260319 16:33:20.721741 30091 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:20.723237 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 30091
I20260319 16:33:20.723367 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:20.723546 25327 external_mini_cluster.cc:1468] Setting key ba206f4d0d461788fb70a5d444731f6b
Mar 19 16:33:20 dist-test-slave-svn8 krb5kdc[29980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938000, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:20.730227 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:20.722348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38103 (local address 127.24.187.193:56879)
0319 16:33:20.722538 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:20.722565 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:20.723007 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:20.723234 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:20.723237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:20.723575 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:20.723929 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:20.723934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.724736 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:20.724736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:20.725116 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:20.725118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.725204 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:20.726080 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:20.726093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:20.727568 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:20.729170 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:20.729175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:20.729177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:20.729351 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:20.729506 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:20.729506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:20.729507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:20.729556 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:20.729816 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:20.729819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:20.729922 (+ 103us) client_negotiation.cc:769] Sending connection context
0319 16:33:20.729999 (+ 77us) client_negotiation.cc:240] Negotiation successful
0319 16:33:20.730094 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
I20260319 16:33:20.730235 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:20.722396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:56879 (local address 127.24.187.254:38103)
0319 16:33:20.722460 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:20.722462 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:20.722634 (+ 172us) server_negotiation.cc:408] Connection header received
0319 16:33:20.723106 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:20.723108 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:20.723120 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:20.723176 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:20.724000 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.724643 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:20.725184 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.725344 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:20.727666 (+ 2322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:20.727670 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:20.727671 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:20.727678 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:20.729092 (+ 1414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:20.729416 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:20.729417 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:20.729418 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:20.729445 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:20.729601 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:20.729602 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:20.729602 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:20.729754 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:20.729821 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:20.730043 (+ 222us) server_negotiation.cc:300] Negotiation successful
0319 16:33:20.730140 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:33:20.730686 30219 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38103
I20260319 16:33:20.730785 30219 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:20.730974 30219 heartbeater.cc:507] Master 127.24.187.254:38103 requested a full tablet report, sending...
I20260319 16:33:20.731474 30027 ts_manager.cc:194] Registered new tserver with Master: 1b72891761c04ebc8b8473437a53d8b0 (127.24.187.193:38543)
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
I20260319 16:33:20.732103 30027 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:56879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:20.768518 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38103
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:20.843545 30226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:20.843706 30226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:20.843736 30226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:20.845134 30226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:20.845177 30226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:20.845211 30226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:20.846815 30226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:20.847023 30226 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:20.847183 30226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:20.849498 30234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:20.849504 30231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:20.849625 30232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:20.849669 30226 server_base.cc:1060] running on GCE node
I20260319 16:33:20.849922 30226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:20.850117 30226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:20.851264 30226 hybrid_clock.cc:648] HybridClock initialized: now 1773938000851245 us; error 26 us; skew 500 ppm
Mar 19 16:33:20 dist-test-slave-svn8 krb5kdc[29980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938000, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:20.853513 30226 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:20.853820 30226 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:20.853865 30226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:20.853967 30226 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:20.854900 30226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "d27203d8ba2349b39bcc7f420b40906c"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "dd0e792ec500bfe1b1b8007cd2c00b4a"
server_key_iv: "95337814787f9675c04c954e849d7e92"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.855290 30226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "d27203d8ba2349b39bcc7f420b40906c"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "dd0e792ec500bfe1b1b8007cd2c00b4a"
server_key_iv: "95337814787f9675c04c954e849d7e92"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.856348 30226 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:20.856946 30240 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.857097 30226 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:20.857153 30226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "d27203d8ba2349b39bcc7f420b40906c"
format_stamp: "Formatted at 2026-03-19 16:33:20 on dist-test-slave-svn8"
server_key: "dd0e792ec500bfe1b1b8007cd2c00b4a"
server_key_iv: "95337814787f9675c04c954e849d7e92"
server_key_version: "encryptionkey@0"
I20260319 16:33:20.857209 30226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:20.871613 30226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:20.873106 30226 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:20.873237 30226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:20.873420 30226 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:20.873733 30226 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:20.873773 30226 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.873802 30226 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:20.873826 30226 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:20.879158 30226 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:46653
I20260319 16:33:20.879204 30353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:46653 every 8 connection(s)
I20260319 16:33:20.879431 30226 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:20.882695 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 30226
I20260319 16:33:20.883013 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:20.883184 25327 external_mini_cluster.cc:1468] Setting key f7245304ef2a95cb9b922a56f8ea2160
Mar 19 16:33:20 dist-test-slave-svn8 krb5kdc[29980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938000, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:20.887660 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:20.880049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:40587 (local address 127.24.187.254:38103)
0319 16:33:20.880089 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:20.880091 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:20.880258 (+ 167us) server_negotiation.cc:408] Connection header received
0319 16:33:20.880730 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:20.880732 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:20.880744 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:20.880798 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:20.881480 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.882027 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:20.882543 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.882712 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:20.885006 (+ 2294us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:20.885010 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:20.885010 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:20.885018 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:20.886304 (+ 1286us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:20.886737 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:20.886738 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:20.886739 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:20.886765 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:20.887003 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:20.887005 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:20.887005 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:20.887126 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:20.887191 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:20.887516 (+ 325us) server_negotiation.cc:300] Negotiation successful
0319 16:33:20.887594 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:33:20.887797 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:20.880010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38103 (local address 127.24.187.194:40587)
0319 16:33:20.880161 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:20.880186 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:20.880641 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:20.880857 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:20.880860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:20.881131 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:20.881383 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:20.881387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.882106 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:20.882107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:20.882471 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:20.882473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:20.882577 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:20.883426 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:20.883440 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:20.884918 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:20.886382 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:20.886388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:20.886390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:20.886631 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:20.886837 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:20.886838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:20.886838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:20.886916 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:20.887195 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:20.887200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:20.887345 (+ 145us) client_negotiation.cc:769] Sending connection context
0319 16:33:20.887465 (+ 120us) client_negotiation.cc:240] Negotiation successful
0319 16:33:20.887603 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260319 16:33:20.888331 30354 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38103
I20260319 16:33:20.888429 30354 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:20.888622 30354 heartbeater.cc:507] Master 127.24.187.254:38103 requested a full tablet report, sending...
I20260319 16:33:20.888999 30027 ts_manager.cc:194] Registered new tserver with Master: d27203d8ba2349b39bcc7f420b40906c (127.24.187.194:46653)
I20260319 16:33:20.889362 30027 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:40587
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:20.928108 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38103
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:21.001238 30361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:21.001410 30361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:21.001449 30361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:21.002914 30361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:21.002957 30361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:21.002990 30361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:21.004562 30361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:38103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:21.004758 30361 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:21.004904 30361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:21.007107 30369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.007141 30367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.007162 30366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.007498 30361 server_base.cc:1060] running on GCE node
I20260319 16:33:21.007671 30361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:21.007849 30361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:21.008972 30361 hybrid_clock.cc:648] HybridClock initialized: now 1773938001008959 us; error 24 us; skew 500 ppm
Mar 19 16:33:21 dist-test-slave-svn8 krb5kdc[29980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938001, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:21.011303 30361 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:21.011623 30361 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:21.011667 30361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:21.011767 30361 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:21.012799 30361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "c630b26c5d0546eb95ec62adc9f1a33c"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "b42967507985a4cc19502760f3e582af"
server_key_iv: "3f284fa0f73375359896c98f7aa15ea3"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.013082 30361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "c630b26c5d0546eb95ec62adc9f1a33c"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "b42967507985a4cc19502760f3e582af"
server_key_iv: "3f284fa0f73375359896c98f7aa15ea3"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.014181 30361 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260319 16:33:21.014817 30375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.014961 30361 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:21.015018 30361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "c630b26c5d0546eb95ec62adc9f1a33c"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "b42967507985a4cc19502760f3e582af"
server_key_iv: "3f284fa0f73375359896c98f7aa15ea3"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.015076 30361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:21.042440 30361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:21.044341 30361 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:21.044464 30361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:21.044644 30361 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:21.044916 30361 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:21.044947 30361 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.044974 30361 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:21.044998 30361 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.050566 30361 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:39011
I20260319 16:33:21.050673 30488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:39011 every 8 connection(s)
I20260319 16:33:21.050834 30361 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:33:21.052596 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 30361
I20260319 16:33:21.052680 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:21.052816 25327 external_mini_cluster.cc:1468] Setting key 9e034d7a53af8ee6337a0d4ad9cfa885
Mar 19 16:33:21 dist-test-slave-svn8 krb5kdc[29980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938001, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:21.058727 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.051510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:37903 (local address 127.24.187.254:38103)
0319 16:33:21.051607 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.051610 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.051693 (+ 83us) server_negotiation.cc:408] Connection header received
0319 16:33:21.052189 (+ 496us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.052191 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.052203 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.052262 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.053076 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.053825 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.054367 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.054498 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.056655 (+ 2157us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.056659 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.056659 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:21.056666 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.057812 (+ 1146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:21.058127 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:21.058128 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:21.058128 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:21.058145 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:21.058267 (+ 122us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:21.058268 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:21.058268 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:21.058346 (+ 78us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.058389 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.058577 (+ 188us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.058653 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260319 16:33:21.058789 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.051442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38103 (local address 127.24.187.195:37903)
0319 16:33:21.051583 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.051620 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.052101 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.052316 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.052320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.052662 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.052989 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.052995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.053908 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.053908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:21.054280 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.054282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.054407 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.055244 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:21.055257 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.056578 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.057876 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:21.057881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:21.057883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:21.058072 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:21.058185 (+ 113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:21.058186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:21.058186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:21.058230 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:21.058383 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.058385 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.058483 (+ 98us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.058559 (+ 76us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.058654 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260319 16:33:21.059228 30489 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38103
I20260319 16:33:21.059304 30489 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:21.059468 30489 heartbeater.cc:507] Master 127.24.187.254:38103 requested a full tablet report, sending...
I20260319 16:33:21.059803 30027 ts_manager.cc:194] Registered new tserver with Master: c630b26c5d0546eb95ec62adc9f1a33c (127.24.187.195:39011)
I20260319 16:33:21.060174 30027 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:37903
I20260319 16:33:21.064410 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:33:21.070775 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.065649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.24.187.254:38103)
0319 16:33:21.065744 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.065746 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.065795 (+ 49us) server_negotiation.cc:408] Connection header received
0319 16:33:21.065859 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.065860 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.065870 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.065913 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.066572 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.067036 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.067542 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.067703 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.068619 (+ 916us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.068623 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.068623 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:21.068631 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.069867 (+ 1236us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:21.070137 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:21.070138 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:21.070139 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:21.070162 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:21.070288 (+ 126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:21.070289 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:21.070289 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:21.070404 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.070520 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.070610 (+ 90us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.070686 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260319 16:33:21.072049 30027 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:21.073246 30027 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:21.078258 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.075471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46653 (local address 127.0.0.1:52656)
0319 16:33:21.075710 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.075718 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.075794 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.076012 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.076014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.076017 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:21.076274 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.076276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.077279 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.077280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:21.077990 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.077991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.078060 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.078102 (+ 42us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.078145 (+ 43us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.078183 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260319 16:33:21.078708 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.075494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:38543 (local address 127.0.0.1:59534)
0319 16:33:21.075914 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.075921 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.075959 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.076163 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.076165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.076167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:21.076382 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.076383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.077766 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.077767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:21.078541 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.078543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.078587 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.078590 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.078610 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.078643 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":90,"threads_started":1}
I20260319 16:33:21.078768 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.075694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.24.187.194:46653)
0319 16:33:21.075859 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.075866 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.075877 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:21.075912 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.075914 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.075926 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.075999 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:21.076390 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.077204 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.078093 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.078571 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.078633 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.078661 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.078704 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260319 16:33:21.079255 30288 tablet_service.cc:1511] Processing CreateTablet for tablet 982aa17f80ea41fda11308347c76942c (DEFAULT_TABLE table=test-table [id=c719a9ea6ab44de082bd4378f682e3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:21.079381 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.075718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.24.187.193:38543)
0319 16:33:21.075859 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.075866 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.075970 (+ 104us) server_negotiation.cc:408] Connection header received
0319 16:33:21.076018 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.076020 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.076034 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.076088 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:21.076568 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.077664 (+ 1096us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.078873 (+ 1209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.079240 (+ 367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.079270 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.079293 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.079318 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260319 16:33:21.079516 30288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982aa17f80ea41fda11308347c76942c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.079792 30153 tablet_service.cc:1511] Processing CreateTablet for tablet 982aa17f80ea41fda11308347c76942c (DEFAULT_TABLE table=test-table [id=c719a9ea6ab44de082bd4378f682e3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:21.080006 30153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982aa17f80ea41fda11308347c76942c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.081801 30506 tablet_bootstrap.cc:492] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Bootstrap starting.
I20260319 16:33:21.081964 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.075455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:39011 (local address 127.0.0.1:57724)
0319 16:33:21.075598 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.075628 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.075857 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.076197 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.076198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.076200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:21.076382 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.076383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.077751 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.077752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:21.078923 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.078926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.081814 (+ 2888us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.081819 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.081849 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.081901 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":44,"threads_started":1}
I20260319 16:33:21.082266 30507 tablet_bootstrap.cc:492] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0: Bootstrap starting.
I20260319 16:33:21.082531 30506 tablet_bootstrap.cc:654] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.082700 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.075524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.24.187.195:39011)
0319 16:33:21.076057 (+ 533us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.076065 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.076075 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:21.076108 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.076110 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.076125 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.076188 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:21.076472 (+ 284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.077640 (+ 1168us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.082092 (+ 4452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.082531 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.082561 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.082590 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.082629 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":71,"threads_started":1}
I20260319 16:33:21.082801 30507 tablet_bootstrap.cc:654] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.082799 30506 log.cc:826] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.083055 30507 log.cc:826] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.083144 30423 tablet_service.cc:1511] Processing CreateTablet for tablet 982aa17f80ea41fda11308347c76942c (DEFAULT_TABLE table=test-table [id=c719a9ea6ab44de082bd4378f682e3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:21.083380 30423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982aa17f80ea41fda11308347c76942c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.083463 30506 tablet_bootstrap.cc:492] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: No bootstrap required, opened a new log
I20260319 16:33:21.083549 30506 ts_tablet_manager.cc:1403] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:21.084782 30506 raft_consensus.cc:359] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.084900 30506 raft_consensus.cc:385] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.084926 30506 raft_consensus.cc:740] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d27203d8ba2349b39bcc7f420b40906c, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.085006 30506 consensus_queue.cc:260] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.085233 30506 ts_tablet_manager.cc:1434] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:21.085340 30354 heartbeater.cc:499] Master 127.24.187.254:38103 was elected leader, sending a full tablet report...
I20260319 16:33:21.085371 30511 tablet_bootstrap.cc:492] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Bootstrap starting.
I20260319 16:33:21.085927 30511 tablet_bootstrap.cc:654] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.086122 30507 tablet_bootstrap.cc:492] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0: No bootstrap required, opened a new log
I20260319 16:33:21.086177 30507 ts_tablet_manager.cc:1403] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260319 16:33:21.086179 30511 log.cc:826] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.086764 30511 tablet_bootstrap.cc:492] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: No bootstrap required, opened a new log
I20260319 16:33:21.086817 30511 ts_tablet_manager.cc:1403] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:21.087509 30507 raft_consensus.cc:359] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.087622 30507 raft_consensus.cc:385] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.087641 30507 raft_consensus.cc:740] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b72891761c04ebc8b8473437a53d8b0, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.087720 30507 consensus_queue.cc:260] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.087908 30507 ts_tablet_manager.cc:1434] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:21.087971 30219 heartbeater.cc:499] Master 127.24.187.254:38103 was elected leader, sending a full tablet report...
I20260319 16:33:21.088001 30511 raft_consensus.cc:359] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.088119 30511 raft_consensus.cc:385] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.088138 30511 raft_consensus.cc:740] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c630b26c5d0546eb95ec62adc9f1a33c, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.088214 30511 consensus_queue.cc:260] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.088588 30511 ts_tablet_manager.cc:1434] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:21.088625 30489 heartbeater.cc:499] Master 127.24.187.254:38103 was elected leader, sending a full tablet report...
I20260319 16:33:21.125063 30513 raft_consensus.cc:493] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:21.125206 30513 raft_consensus.cc:515] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.125571 30513 leader_election.cc:290] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c630b26c5d0546eb95ec62adc9f1a33c (127.24.187.195:39011), d27203d8ba2349b39bcc7f420b40906c (127.24.187.194:46653)
I20260319 16:33:21.127662 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.125776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:39011 (local address 127.24.187.193:60763)
0319 16:33:21.125832 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.125845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.125886 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.126046 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.126047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.126049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:21.126251 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.126253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.126917 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.126918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:21.127508 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.127510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.127566 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.127569 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.127589 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.127613 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260319 16:33:21.127837 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.125790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46653 (local address 127.24.187.193:59005)
0319 16:33:21.125959 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.125970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.126030 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.126147 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.126147 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.126149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:21.126373 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.126374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.127016 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.127017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:21.127631 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:21.127633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.127703 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.127707 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.127735 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.127767 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260319 16:33:21.128243 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.125830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:60763 (local address 127.24.187.195:39011)
0319 16:33:21.125876 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.125878 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.125888 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:21.125956 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.125957 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.125967 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.126035 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:21.126316 (+ 281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.126842 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.127584 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.128150 (+ 566us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.128157 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.128177 (+ 20us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.128200 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:33:21.128311 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.125842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:59005 (local address 127.24.187.194:46653)
0319 16:33:21.125897 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.125899 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.126036 (+ 137us) server_negotiation.cc:408] Connection header received
0319 16:33:21.126072 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.126074 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.126083 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.126140 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:21.126456 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.126943 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.127722 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.128212 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.128220 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.128239 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.128261 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:33:21.128525 30443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982aa17f80ea41fda11308347c76942c" candidate_uuid: "1b72891761c04ebc8b8473437a53d8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" is_pre_election: true
I20260319 16:33:21.128577 30308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982aa17f80ea41fda11308347c76942c" candidate_uuid: "1b72891761c04ebc8b8473437a53d8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d27203d8ba2349b39bcc7f420b40906c" is_pre_election: true
I20260319 16:33:21.128691 30308 raft_consensus.cc:2468] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b72891761c04ebc8b8473437a53d8b0 in term 0.
I20260319 16:33:21.128695 30443 raft_consensus.cc:2468] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b72891761c04ebc8b8473437a53d8b0 in term 0.
I20260319 16:33:21.128912 30108 leader_election.cc:304] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b72891761c04ebc8b8473437a53d8b0, d27203d8ba2349b39bcc7f420b40906c; no voters:
I20260319 16:33:21.129041 30513 raft_consensus.cc:2804] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:21.129076 30513 raft_consensus.cc:493] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:21.129099 30513 raft_consensus.cc:3060] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.129796 30513 raft_consensus.cc:515] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
W20260319 16:33:21.129875 30355 tablet.cc:2379] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:33:21.129914 30513 leader_election.cc:290] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [CANDIDATE]: Term 1 election: Requested vote from peers c630b26c5d0546eb95ec62adc9f1a33c (127.24.187.195:39011), d27203d8ba2349b39bcc7f420b40906c (127.24.187.194:46653)
I20260319 16:33:21.130088 30443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982aa17f80ea41fda11308347c76942c" candidate_uuid: "1b72891761c04ebc8b8473437a53d8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c630b26c5d0546eb95ec62adc9f1a33c"
I20260319 16:33:21.130088 30308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982aa17f80ea41fda11308347c76942c" candidate_uuid: "1b72891761c04ebc8b8473437a53d8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d27203d8ba2349b39bcc7f420b40906c"
I20260319 16:33:21.130163 30308 raft_consensus.cc:3060] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.130163 30443 raft_consensus.cc:3060] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.130985 30308 raft_consensus.cc:2468] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b72891761c04ebc8b8473437a53d8b0 in term 1.
I20260319 16:33:21.130985 30443 raft_consensus.cc:2468] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b72891761c04ebc8b8473437a53d8b0 in term 1.
I20260319 16:33:21.131145 30108 leader_election.cc:304] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b72891761c04ebc8b8473437a53d8b0, d27203d8ba2349b39bcc7f420b40906c; no voters:
I20260319 16:33:21.131242 30513 raft_consensus.cc:2804] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:21.131376 30513 raft_consensus.cc:697] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 1 LEADER]: Becoming Leader. State: Replica: 1b72891761c04ebc8b8473437a53d8b0, State: Running, Role: LEADER
I20260319 16:33:21.131456 30513 consensus_queue.cc:237] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } }
I20260319 16:33:21.132206 30027 catalog_manager.cc:5671] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b72891761c04ebc8b8473437a53d8b0 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "1b72891761c04ebc8b8473437a53d8b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b72891761c04ebc8b8473437a53d8b0" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38543 } health_report { overall_health: HEALTHY } } }
I20260319 16:33:21.136600 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.134792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.24.187.193:38543)
0319 16:33:21.134853 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.134855 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.134863 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:21.134897 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.134898 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.134915 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.134964 (+ 49us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:21.135178 (+ 214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.135633 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.136167 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.136301 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.136334 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:21.136415 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:21.136478 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.136517 (+ 39us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.136540 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260319 16:33:21.137995 30443 raft_consensus.cc:1275] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 1 FOLLOWER]: Refusing update from remote peer 1b72891761c04ebc8b8473437a53d8b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:21.137995 30308 raft_consensus.cc:1275] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 1 FOLLOWER]: Refusing update from remote peer 1b72891761c04ebc8b8473437a53d8b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:21.138260 30516 consensus_queue.cc:1048] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d27203d8ba2349b39bcc7f420b40906c" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:21.138363 30513 consensus_queue.cc:1048] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c630b26c5d0546eb95ec62adc9f1a33c" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:21.140442 30519 mvcc.cc:204] Tried to move back new op lower bound from 7266050052658946048 to 7266050052634963968. Current Snapshot: MvccSnapshot[applied={T|T < 7266050052658946048}]
I20260319 16:33:21.141006 30520 mvcc.cc:204] Tried to move back new op lower bound from 7266050052658946048 to 7266050052634963968. Current Snapshot: MvccSnapshot[applied={T|T < 7266050052658946048}]
I20260319 16:33:21.141216 30521 mvcc.cc:204] Tried to move back new op lower bound from 7266050052658946048 to 7266050052634963968. Current Snapshot: MvccSnapshot[applied={T|T < 7266050052658946048 or (T in {7266050052658946048})}]
I20260319 16:33:21.142462 30027 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:21.142542 30027 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:21.143177 30027 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1f2b243e495641a98e448cb4c2bc8c6b: Sending DeleteTablet for 3 replicas of tablet 982aa17f80ea41fda11308347c76942c
I20260319 16:33:21.143505 30288 tablet_service.cc:1558] Processing DeleteTablet for tablet 982aa17f80ea41fda11308347c76942c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:21 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:52656
I20260319 16:33:21.143553 30423 tablet_service.cc:1558] Processing DeleteTablet for tablet 982aa17f80ea41fda11308347c76942c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:21 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:57724
I20260319 16:33:21.143728 30529 tablet_replica.cc:333] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: stopping tablet replica
I20260319 16:33:21.143805 30529 raft_consensus.cc:2243] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:21.143788 30153 tablet_service.cc:1558] Processing DeleteTablet for tablet 982aa17f80ea41fda11308347c76942c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:21 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:59534
I20260319 16:33:21.143900 30529 raft_consensus.cc:2272] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:21.143980 30531 tablet_replica.cc:333] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0: stopping tablet replica
I20260319 16:33:21.144086 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 30091
I20260319 16:33:21.144098 30531 raft_consensus.cc:2243] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:21.144202 30531 raft_consensus.cc:2272] T 982aa17f80ea41fda11308347c76942c P 1b72891761c04ebc8b8473437a53d8b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:21.144361 30529 ts_tablet_manager.cc:1916] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:21.146145 30529 ts_tablet_manager.cc:1929] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:21.146201 30529 log.cc:1199] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/982aa17f80ea41fda11308347c76942c
I20260319 16:33:21.146461 30529 ts_tablet_manager.cc:1950] T 982aa17f80ea41fda11308347c76942c P d27203d8ba2349b39bcc7f420b40906c: Deleting consensus metadata
I20260319 16:33:21.146648 30530 tablet_replica.cc:333] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: stopping tablet replica
I20260319 16:33:21.146775 30530 raft_consensus.cc:2243] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:21.146870 30530 raft_consensus.cc:2272] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:21.147058 30013 catalog_manager.cc:5002] TS d27203d8ba2349b39bcc7f420b40906c (127.24.187.194:46653): tablet 982aa17f80ea41fda11308347c76942c (table test-table [id=c719a9ea6ab44de082bd4378f682e3d7]) successfully deleted
I20260319 16:33:21.147363 30530 ts_tablet_manager.cc:1916] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:21.148818 30530 ts_tablet_manager.cc:1929] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:21.148885 30530 log.cc:1199] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/982aa17f80ea41fda11308347c76942c
I20260319 16:33:21.149117 30530 ts_tablet_manager.cc:1950] T 982aa17f80ea41fda11308347c76942c P c630b26c5d0546eb95ec62adc9f1a33c: Deleting consensus metadata
I20260319 16:33:21.149519 30012 catalog_manager.cc:5002] TS c630b26c5d0546eb95ec62adc9f1a33c (127.24.187.195:39011): tablet 982aa17f80ea41fda11308347c76942c (table test-table [id=c719a9ea6ab44de082bd4378f682e3d7]) successfully deleted
W20260319 16:33:21.150586 30011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:38543 (error 108)
I20260319 16:33:21.150821 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 30226
I20260319 16:33:21.151050 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.150949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:38543 (local address 127.0.0.1:59542)
0319 16:33:21.150993 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.151019 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260319 16:33:21.151124 30011 catalog_manager.cc:4729] TS 1b72891761c04ebc8b8473437a53d8b0 (127.24.187.193:38543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 982aa17f80ea41fda11308347c76942c: Network error: Client connection negotiation failed: client connection to 127.24.187.193:38543: connect: Connection refused (error 111)
I20260319 16:33:21.156072 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 30361
I20260319 16:33:21.160566 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 29996
2026-03-19T16:33:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2709 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-03-19T16:33:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:21Z Disabled control of system clock
I20260319 16:33:21.186385 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:45939
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:45939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260319 16:33:21.260320 30536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:21.260483 30536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:21.260502 30536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:21.261955 30536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:21.261997 30536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:21.262007 30536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:21.262019 30536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:21.262029 30536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:21.263576 30536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:45939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:21.263767 30536 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:21.263939 30536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:21.266124 30544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.266129 30541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.266122 30542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.266368 30536 server_base.cc:1060] running on GCE node
I20260319 16:33:21.266628 30536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:21.266856 30536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:21.267992 30536 hybrid_clock.cc:648] HybridClock initialized: now 1773938001267972 us; error 27 us; skew 500 ppm
I20260319 16:33:21.269057 30536 webserver.cc:492] Webserver started at http://127.24.187.254:41297/ using document root <none> and password file <none>
I20260319 16:33:21.269232 30536 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:21.269271 30536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:21.269375 30536 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:21.270360 30536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "7250864a4c65481e84f1ec48f1145908"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "98260fbf5c185495da77bcdfe80009ec"
server_key_iv: "1e250b2fcda6748a36bec1ce0758dc75"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.270654 30536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "7250864a4c65481e84f1ec48f1145908"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "98260fbf5c185495da77bcdfe80009ec"
server_key_iv: "1e250b2fcda6748a36bec1ce0758dc75"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.271687 30536 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:21.272306 30550 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.272491 30536 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:21.272557 30536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "7250864a4c65481e84f1ec48f1145908"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "98260fbf5c185495da77bcdfe80009ec"
server_key_iv: "1e250b2fcda6748a36bec1ce0758dc75"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.272622 30536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:21.275832 30536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:21.276019 30536 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:21.276114 30536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:21.279651 30536 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:45939
I20260319 16:33:21.279696 30602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:45939 every 8 connection(s)
I20260319 16:33:21.279959 30536 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:21.280534 30603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.282816 30603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908: Bootstrap starting.
I20260319 16:33:21.283367 30603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.283576 30603 log.cc:826] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.284204 30603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908: No bootstrap required, opened a new log
I20260319 16:33:21.285429 30603 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7250864a4c65481e84f1ec48f1145908" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45939 } }
I20260319 16:33:21.285545 30603 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.285573 30603 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7250864a4c65481e84f1ec48f1145908, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.285681 30603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7250864a4c65481e84f1ec48f1145908" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45939 } }
I20260319 16:33:21.285775 30603 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:21.285805 30603 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:21.285848 30603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.286386 30603 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7250864a4c65481e84f1ec48f1145908" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45939 } }
I20260319 16:33:21.286485 30603 leader_election.cc:304] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7250864a4c65481e84f1ec48f1145908; no voters:
I20260319 16:33:21.286615 30603 leader_election.cc:290] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:21.286705 30606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:21.286834 30603 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:21.286847 30606 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [term 1 LEADER]: Becoming Leader. State: Replica: 7250864a4c65481e84f1ec48f1145908, State: Running, Role: LEADER
I20260319 16:33:21.287041 30606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7250864a4c65481e84f1ec48f1145908" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45939 } }
I20260319 16:33:21.287407 30608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7250864a4c65481e84f1ec48f1145908. Latest consensus state: current_term: 1 leader_uuid: "7250864a4c65481e84f1ec48f1145908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7250864a4c65481e84f1ec48f1145908" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45939 } } }
I20260319 16:33:21.287418 30607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7250864a4c65481e84f1ec48f1145908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7250864a4c65481e84f1ec48f1145908" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45939 } } }
I20260319 16:33:21.287545 30608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:21.287555 30607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:21.288069 30615 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:21.288447 30615 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:21.289693 30615 catalog_manager.cc:1357] Generated new cluster ID: 6f3e2676d7ac4f0faadaf7704600b050
I20260319 16:33:21.289767 30615 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:21.289903 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 30536
I20260319 16:33:21.290005 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:21.290171 25327 external_mini_cluster.cc:1468] Setting key b20c259576327ebff05d96f5c22a23c6
I20260319 16:33:21.292115 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.290607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.24.187.254:45939)
0319 16:33:21.290797 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.290804 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.290827 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:33:21.290922 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.290928 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.290971 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.291083 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.291692 (+ 609us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.291695 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.291696 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.291699 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.291724 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.291730 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.291794 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.291861 (+ 67us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.291984 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260319 16:33:21.299870 30615 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:21.299997 30615 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:21.306597 30615 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908: Generated new TSK 0
I20260319 16:33:21.306756 30615 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:33:21.310182 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45939
--builtin_ntp_servers=127.24.187.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260319 16:33:21.384769 30627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:21.384929 30627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:21.384958 30627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:21.386421 30627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:21.386462 30627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:21.386494 30627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:21.388034 30627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:21.388250 30627 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:21.388459 30627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:21.390880 30633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.390892 30632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.391119 30635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.391259 30627 server_base.cc:1060] running on GCE node
I20260319 16:33:21.391417 30627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:21.391631 30627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:21.392758 30627 hybrid_clock.cc:648] HybridClock initialized: now 1773938001392733 us; error 39 us; skew 500 ppm
I20260319 16:33:21.393805 30627 webserver.cc:492] Webserver started at http://127.24.187.193:36881/ using document root <none> and password file <none>
I20260319 16:33:21.393972 30627 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:21.394012 30627 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:21.394114 30627 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:21.395051 30627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "951f1a3d70a94c0d9122b4cea7814007"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "968c5967ea27b28e8e3fdc57251e24a6"
server_key_iv: "b4b47d690274107383ef3ed0578bc699"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.395339 30627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "951f1a3d70a94c0d9122b4cea7814007"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "968c5967ea27b28e8e3fdc57251e24a6"
server_key_iv: "b4b47d690274107383ef3ed0578bc699"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.396610 30627 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260319 16:33:21.397275 30641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.397442 30627 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:21.397497 30627 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "951f1a3d70a94c0d9122b4cea7814007"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "968c5967ea27b28e8e3fdc57251e24a6"
server_key_iv: "b4b47d690274107383ef3ed0578bc699"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.397557 30627 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:21.400697 30627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:21.400872 30627 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:21.400965 30627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:21.401134 30627 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:21.401389 30627 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:21.401418 30627 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.401451 30627 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:21.401475 30627 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.407487 30627 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:45979
I20260319 16:33:21.407552 30754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:45979 every 8 connection(s)
I20260319 16:33:21.407804 30627 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:21.410347 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.408469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:34775 (local address 127.24.187.254:45939)
0319 16:33:21.408516 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.408518 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.408698 (+ 180us) server_negotiation.cc:408] Connection header received
0319 16:33:21.409291 (+ 593us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.409292 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.409296 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.409345 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.410108 (+ 763us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.410110 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.410110 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.410114 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.410130 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.410133 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.410188 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.410281 (+ 93us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.410294 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:33:21.410499 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.408382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45939 (local address 127.24.187.193:34775)
0319 16:33:21.408564 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.408596 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.409183 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.409399 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.409403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.409414 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.409945 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.409946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.409969 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.409990 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.410199 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.410202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.410206 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.410280 (+ 74us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.410329 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260319 16:33:21.411005 30755 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45939
I20260319 16:33:21.411091 30755 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:21.411212 30755 heartbeater.cc:507] Master 127.24.187.254:45939 requested a full tablet report, sending...
I20260319 16:33:21.411746 30567 ts_manager.cc:194] Registered new tserver with Master: 951f1a3d70a94c0d9122b4cea7814007 (127.24.187.193:45979)
I20260319 16:33:21.414603 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 30627
I20260319 16:33:21.414711 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:21.414880 25327 external_mini_cluster.cc:1468] Setting key bca6734dc00d98a4a415f67d0f340e8c
I20260319 16:33:21.415756 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45939
--builtin_ntp_servers=127.24.187.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260319 16:33:21.489874 30758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:21.490020 30758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:21.490039 30758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:21.491375 30758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:21.491415 30758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:21.491437 30758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:21.492913 30758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:21.493111 30758 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:21.493345 30758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:21.495723 30763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.495714 30766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.495738 30764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.495965 30758 server_base.cc:1060] running on GCE node
I20260319 16:33:21.496130 30758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:21.496306 30758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:21.497442 30758 hybrid_clock.cc:648] HybridClock initialized: now 1773938001497422 us; error 35 us; skew 500 ppm
I20260319 16:33:21.498533 30758 webserver.cc:492] Webserver started at http://127.24.187.194:34713/ using document root <none> and password file <none>
I20260319 16:33:21.498718 30758 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:21.498757 30758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:21.498855 30758 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:21.499744 30758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "0fe13408ff5b8177c6e72df9ccf5b96f"
server_key_iv: "72daa56cd539e2c6ed7bbe68700b7aa8"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.500025 30758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "0fe13408ff5b8177c6e72df9ccf5b96f"
server_key_iv: "72daa56cd539e2c6ed7bbe68700b7aa8"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.501091 30758 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:33:21.501868 30772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.502018 30758 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:21.502082 30758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "0fe13408ff5b8177c6e72df9ccf5b96f"
server_key_iv: "72daa56cd539e2c6ed7bbe68700b7aa8"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.502143 30758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:21.505301 30758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:21.505473 30758 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:21.505568 30758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:21.505760 30758 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:21.506018 30758 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:21.506047 30758 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.506071 30758 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:21.506098 30758 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.511420 30758 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:32883
I20260319 16:33:21.511469 30885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:32883 every 8 connection(s)
I20260319 16:33:21.511708 30758 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:21.513993 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.512313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:40775 (local address 127.24.187.254:45939)
0319 16:33:21.512394 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.512396 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.512523 (+ 127us) server_negotiation.cc:408] Connection header received
0319 16:33:21.512937 (+ 414us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.512938 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.512945 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.513001 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.513792 (+ 791us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.513794 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.513795 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.513799 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.513815 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.513819 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.513867 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.513923 (+ 56us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.513937 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:33:21.514101 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.512280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45939 (local address 127.24.187.194:40775)
0319 16:33:21.512450 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.512474 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.512870 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.513160 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.513165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.513174 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.513661 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.513662 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.513678 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.513695 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.513870 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.513873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.513875 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.513924 (+ 49us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.513960 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260319 16:33:21.514616 30886 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45939
I20260319 16:33:21.514693 30886 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:21.514797 30886 heartbeater.cc:507] Master 127.24.187.254:45939 requested a full tablet report, sending...
I20260319 16:33:21.515175 30567 ts_manager.cc:194] Registered new tserver with Master: 9fb8491cb29f4155b31fc7f35e2d4bf3 (127.24.187.194:32883)
I20260319 16:33:21.519984 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 30758
I20260319 16:33:21.520076 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:21.520238 25327 external_mini_cluster.cc:1468] Setting key 25cb1e22d571ab5deccd07d3e6df9345
I20260319 16:33:21.521085 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45939
--builtin_ntp_servers=127.24.187.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260319 16:33:21.595541 30889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:21.595688 30889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:21.595706 30889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:21.597049 30889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:21.597085 30889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:21.597107 30889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:21.598593 30889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:21.598758 30889 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:21.598907 30889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:21.601199 30894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.601207 30895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.601322 30889 server_base.cc:1060] running on GCE node
W20260319 16:33:21.601444 30897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.601613 30889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:21.601944 30889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:21.603083 30889 hybrid_clock.cc:648] HybridClock initialized: now 1773938001603064 us; error 31 us; skew 500 ppm
I20260319 16:33:21.604113 30889 webserver.cc:492] Webserver started at http://127.24.187.195:43627/ using document root <none> and password file <none>
I20260319 16:33:21.604279 30889 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:21.604316 30889 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:21.604410 30889 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:21.605346 30889 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "b8ec04da1a74436ca07c0fe771321da3"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "72b5479974dd446d7cb49e8a6351efb2"
server_key_iv: "d8eefa0bb3ace0d09b8845bbbc580bb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.605676 30889 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "b8ec04da1a74436ca07c0fe771321da3"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "72b5479974dd446d7cb49e8a6351efb2"
server_key_iv: "d8eefa0bb3ace0d09b8845bbbc580bb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.606730 30889 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:33:21.607335 30903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.607484 30889 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:21.607539 30889 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "b8ec04da1a74436ca07c0fe771321da3"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "72b5479974dd446d7cb49e8a6351efb2"
server_key_iv: "d8eefa0bb3ace0d09b8845bbbc580bb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.607599 30889 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:21.610760 30889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:21.610929 30889 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:21.611008 30889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:21.611151 30889 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:21.611408 30889 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:21.611444 30889 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.611469 30889 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:21.611492 30889 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.616794 30889 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:37203
I20260319 16:33:21.616855 31016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:37203 every 8 connection(s)
I20260319 16:33:21.617096 30889 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:33:21.619376 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.617779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:43417 (local address 127.24.187.254:45939)
0319 16:33:21.617832 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.617835 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.618002 (+ 167us) server_negotiation.cc:408] Connection header received
0319 16:33:21.618452 (+ 450us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.618454 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.618460 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.618499 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.619151 (+ 652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.619152 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.619152 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.619157 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.619173 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.619177 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.619222 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.619294 (+ 72us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.619309 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:21.619513 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.617742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45939 (local address 127.24.187.195:43417)
0319 16:33:21.617911 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.617936 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.618368 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.618563 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.618566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.618574 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.619025 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.619026 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.619042 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.619061 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.619226 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.619229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.619231 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.619297 (+ 66us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.619345 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260319 16:33:21.619927 31017 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45939
I20260319 16:33:21.620012 31017 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:21.620100 31017 heartbeater.cc:507] Master 127.24.187.254:45939 requested a full tablet report, sending...
I20260319 16:33:21.620406 30567 ts_manager.cc:194] Registered new tserver with Master: b8ec04da1a74436ca07c0fe771321da3 (127.24.187.195:37203)
I20260319 16:33:21.625156 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 30889
I20260319 16:33:21.625240 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:21.625392 25327 external_mini_cluster.cc:1468] Setting key 589f6db35ef76e47569eb4a0497bc598
I20260319 16:33:21.626223 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:33:21.628399 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.627536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.24.187.254:45939)
0319 16:33:21.627581 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.627582 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.627724 (+ 142us) server_negotiation.cc:408] Connection header received
0319 16:33:21.627752 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.627753 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.627758 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.627806 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.628249 (+ 443us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.628251 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.628251 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.628254 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.628268 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.628272 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.628310 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.628338 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.628349 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:33:21.629824 30567 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:21.631057 30567 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:21.634747 31029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.633143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.24.187.195:37203)
0319 16:33:21.633306 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.633311 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.633397 (+ 86us) server_negotiation.cc:408] Connection header received
0319 16:33:21.633449 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.633452 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.633469 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.633537 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.634192 (+ 655us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.634196 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.634197 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.634201 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.634247 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.634253 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.634318 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.634460 (+ 142us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.634474 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260319 16:33:21.634931 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.632960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:32883 (local address 127.0.0.1:36582)
0319 16:33:21.633518 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.633526 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.633561 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.633998 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.633999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.634002 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.634373 (+ 371us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.634374 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.634381 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.634392 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.634587 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.634588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.634589 (+ 1us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.634856 (+ 267us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.634871 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":78,"threads_started":1}
I20260319 16:33:21.635103 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.632943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:37203 (local address 127.0.0.1:38416)
0319 16:33:21.633337 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.633345 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.633393 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.633553 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.633558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.633565 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.634049 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.634050 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.634080 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.634100 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.634364 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.634369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.634372 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.634898 (+ 526us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.634911 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":99,"threads_started":1}
I20260319 16:33:21.635162 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.633155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.24.187.194:32883)
0319 16:33:21.633359 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.633397 (+ 38us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.633614 (+ 217us) server_negotiation.cc:408] Connection header received
0319 16:33:21.633641 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.633643 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.633651 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.633732 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.634473 (+ 741us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.634480 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.634480 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.634485 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.634526 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.634531 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.634605 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.634645 (+ 40us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.634659 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
I20260319 16:33:21.635241 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.632936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:45979 (local address 127.0.0.1:56102)
0319 16:33:21.633179 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.633224 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.633447 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.634514 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.634514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.634517 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.634875 (+ 358us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.634875 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.634882 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.634894 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.635149 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.635150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.635151 (+ 1us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.635192 (+ 41us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.635204 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":56,"threads_started":1}
I20260319 16:33:21.635483 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.633016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.24.187.193:45979)
0319 16:33:21.633184 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.633207 (+ 23us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.633290 (+ 83us) server_negotiation.cc:408] Connection header received
0319 16:33:21.633506 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.633510 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.633519 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.633576 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.634973 (+ 1397us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.634978 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.634978 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.634982 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.635029 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.635035 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.635122 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.635387 (+ 265us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.635401 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260319 16:33:21.635763 30820 tablet_service.cc:1511] Processing CreateTablet for tablet 02ad4a66fa10454fabc3da9a304cc756 (DEFAULT_TABLE table=test-table [id=5c05d2859e204aa198a1da8a49a54cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:21.636023 30820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ad4a66fa10454fabc3da9a304cc756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.636001 30687 tablet_service.cc:1511] Processing CreateTablet for tablet 02ad4a66fa10454fabc3da9a304cc756 (DEFAULT_TABLE table=test-table [id=5c05d2859e204aa198a1da8a49a54cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:21.636219 30687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ad4a66fa10454fabc3da9a304cc756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.638070 30951 tablet_service.cc:1511] Processing CreateTablet for tablet 02ad4a66fa10454fabc3da9a304cc756 (DEFAULT_TABLE table=test-table [id=5c05d2859e204aa198a1da8a49a54cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:21.638253 31031 tablet_bootstrap.cc:492] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Bootstrap starting.
I20260319 16:33:21.638314 30951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ad4a66fa10454fabc3da9a304cc756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.638589 31032 tablet_bootstrap.cc:492] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: Bootstrap starting.
I20260319 16:33:21.638962 31031 tablet_bootstrap.cc:654] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.639137 31032 tablet_bootstrap.cc:654] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.639236 31031 log.cc:826] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.639384 31032 log.cc:826] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.639956 31031 tablet_bootstrap.cc:492] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: No bootstrap required, opened a new log
I20260319 16:33:21.640013 31031 ts_tablet_manager.cc:1403] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:21.640021 31032 tablet_bootstrap.cc:492] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: No bootstrap required, opened a new log
I20260319 16:33:21.640072 31032 ts_tablet_manager.cc:1403] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:21.640715 31035 tablet_bootstrap.cc:492] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Bootstrap starting.
I20260319 16:33:21.641283 31035 tablet_bootstrap.cc:654] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.641402 31032 raft_consensus.cc:359] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.641517 31032 raft_consensus.cc:385] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.641544 31032 raft_consensus.cc:740] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951f1a3d70a94c0d9122b4cea7814007, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.641458 31031 raft_consensus.cc:359] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.641559 31035 log.cc:826] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.641577 31031 raft_consensus.cc:385] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.641603 31031 raft_consensus.cc:740] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb8491cb29f4155b31fc7f35e2d4bf3, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.641625 31032 consensus_queue.cc:260] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.641683 31031 consensus_queue.cc:260] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.641870 31032 ts_tablet_manager.cc:1434] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:33:21.641913 31031 ts_tablet_manager.cc:1434] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:21.641981 30755 heartbeater.cc:499] Master 127.24.187.254:45939 was elected leader, sending a full tablet report...
I20260319 16:33:21.642027 30886 heartbeater.cc:499] Master 127.24.187.254:45939 was elected leader, sending a full tablet report...
I20260319 16:33:21.642282 31035 tablet_bootstrap.cc:492] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: No bootstrap required, opened a new log
I20260319 16:33:21.642330 31035 ts_tablet_manager.cc:1403] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260319 16:33:21.643571 31035 raft_consensus.cc:359] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.643658 31035 raft_consensus.cc:385] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.643673 31035 raft_consensus.cc:740] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ec04da1a74436ca07c0fe771321da3, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.643735 31035 consensus_queue.cc:260] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.643885 31035 ts_tablet_manager.cc:1434] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:33:21.643940 31017 heartbeater.cc:499] Master 127.24.187.254:45939 was elected leader, sending a full tablet report...
W20260319 16:33:21.658352 30756 tablet.cc:2379] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:33:21.762187 30887 tablet.cc:2379] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:33:21.764019 31038 raft_consensus.cc:493] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:21.764123 31038 raft_consensus.cc:515] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.764518 31038 leader_election.cc:290] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 951f1a3d70a94c0d9122b4cea7814007 (127.24.187.193:45979), b8ec04da1a74436ca07c0fe771321da3 (127.24.187.195:37203)
I20260319 16:33:21.765759 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.764712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:45979 (local address 127.24.187.194:40807)
0319 16:33:21.764779 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.764790 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.764829 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.764988 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.764989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.764991 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.765412 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.765413 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.765420 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.765430 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.765625 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.765625 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.765626 (+ 1us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.765669 (+ 43us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.765680 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260319 16:33:21.765799 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.764798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:40807 (local address 127.24.187.193:45979)
0319 16:33:21.764848 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.764850 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.764862 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:33:21.764896 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.764897 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.764905 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.764972 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.765519 (+ 547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.765522 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.765522 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.765528 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.765550 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.765554 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.765617 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.765681 (+ 64us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.765700 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:21.765959 31029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.764802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:57017 (local address 127.24.187.195:37203)
0319 16:33:21.764848 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.764850 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.764985 (+ 135us) server_negotiation.cc:408] Connection header received
0319 16:33:21.765019 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.765020 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.765027 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.765078 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.765651 (+ 573us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.765652 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.765653 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.765657 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.765681 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.765686 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.765787 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.765872 (+ 85us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.765891 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:21.766110 30696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ad4a66fa10454fabc3da9a304cc756" candidate_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951f1a3d70a94c0d9122b4cea7814007" is_pre_election: true
I20260319 16:33:21.766275 30696 raft_consensus.cc:2468] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb8491cb29f4155b31fc7f35e2d4bf3 in term 0.
I20260319 16:33:21.766302 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.764730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:37203 (local address 127.24.187.194:57017)
0319 16:33:21.764910 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.764918 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:21.764970 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:21.765105 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:21.765106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:21.765107 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:21.765561 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:33:21.765561 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:21.765566 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:33:21.765573 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:21.765789 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:21.765789 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:21.765790 (+ 1us) client_negotiation.cc:769] Sending connection context
0319 16:33:21.766063 (+ 273us) client_negotiation.cc:240] Negotiation successful
0319 16:33:21.766076 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":132,"threads_started":1}
I20260319 16:33:21.766502 30776 leader_election.cc:304] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 951f1a3d70a94c0d9122b4cea7814007, 9fb8491cb29f4155b31fc7f35e2d4bf3; no voters:
I20260319 16:33:21.766583 30971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ad4a66fa10454fabc3da9a304cc756" candidate_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ec04da1a74436ca07c0fe771321da3" is_pre_election: true
I20260319 16:33:21.766626 31038 raft_consensus.cc:2804] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:21.766705 31038 raft_consensus.cc:493] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:21.766708 30971 raft_consensus.cc:2468] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb8491cb29f4155b31fc7f35e2d4bf3 in term 0.
I20260319 16:33:21.766738 31038 raft_consensus.cc:3060] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.767501 31038 raft_consensus.cc:515] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.767635 31038 leader_election.cc:290] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [CANDIDATE]: Term 1 election: Requested vote from peers 951f1a3d70a94c0d9122b4cea7814007 (127.24.187.193:45979), b8ec04da1a74436ca07c0fe771321da3 (127.24.187.195:37203)
I20260319 16:33:21.767853 30696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ad4a66fa10454fabc3da9a304cc756" candidate_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951f1a3d70a94c0d9122b4cea7814007"
I20260319 16:33:21.767853 30971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02ad4a66fa10454fabc3da9a304cc756" candidate_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ec04da1a74436ca07c0fe771321da3"
I20260319 16:33:21.767938 30971 raft_consensus.cc:3060] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.767938 30696 raft_consensus.cc:3060] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.769196 30696 raft_consensus.cc:2468] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb8491cb29f4155b31fc7f35e2d4bf3 in term 1.
I20260319 16:33:21.769196 30971 raft_consensus.cc:2468] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb8491cb29f4155b31fc7f35e2d4bf3 in term 1.
I20260319 16:33:21.769441 30775 leader_election.cc:304] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fb8491cb29f4155b31fc7f35e2d4bf3, b8ec04da1a74436ca07c0fe771321da3; no voters:
I20260319 16:33:21.769557 31038 raft_consensus.cc:2804] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:21.769701 31038 raft_consensus.cc:697] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 1 LEADER]: Becoming Leader. State: Replica: 9fb8491cb29f4155b31fc7f35e2d4bf3, State: Running, Role: LEADER
I20260319 16:33:21.769822 31038 consensus_queue.cc:237] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } }
I20260319 16:33:21.770553 30566 catalog_manager.cc:5671] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fb8491cb29f4155b31fc7f35e2d4bf3 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb8491cb29f4155b31fc7f35e2d4bf3" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 32883 } health_report { overall_health: HEALTHY } } }
I20260319 16:33:21.781234 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.780321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.24.187.194:32883)
0319 16:33:21.780375 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.780376 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.780385 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:21.780463 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.780464 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.780474 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.780520 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.781014 (+ 494us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.781015 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.781016 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.781020 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.781035 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.781039 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.781086 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.781147 (+ 61us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.781162 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:33:21.782788 30971 raft_consensus.cc:1275] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 1 FOLLOWER]: Refusing update from remote peer 9fb8491cb29f4155b31fc7f35e2d4bf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:21.782789 30696 raft_consensus.cc:1275] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 1 FOLLOWER]: Refusing update from remote peer 9fb8491cb29f4155b31fc7f35e2d4bf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:21.783084 31041 consensus_queue.cc:1048] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ec04da1a74436ca07c0fe771321da3" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:21.783190 31038 consensus_queue.cc:1048] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "951f1a3d70a94c0d9122b4cea7814007" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:21.785455 31045 mvcc.cc:204] Tried to move back new op lower bound from 7266050055299907584 to 7266050055249694720. Current Snapshot: MvccSnapshot[applied={T|T < 7266050055299907584}]
I20260319 16:33:21.785617 31046 mvcc.cc:204] Tried to move back new op lower bound from 7266050055299907584 to 7266050055249694720. Current Snapshot: MvccSnapshot[applied={T|T < 7266050055299907584}]
I20260319 16:33:21.787708 30566 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:21.787791 30566 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:21.788407 30566 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 7250864a4c65481e84f1ec48f1145908: Sending DeleteTablet for 3 replicas of tablet 02ad4a66fa10454fabc3da9a304cc756
I20260319 16:33:21.788712 30951 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ad4a66fa10454fabc3da9a304cc756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:21 UTC) from {username='slave'} at 127.0.0.1:38416
I20260319 16:33:21.788708 30687 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ad4a66fa10454fabc3da9a304cc756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:21 UTC) from {username='slave'} at 127.0.0.1:56102
I20260319 16:33:21.788815 30820 tablet_service.cc:1558] Processing DeleteTablet for tablet 02ad4a66fa10454fabc3da9a304cc756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:21 UTC) from {username='slave'} at 127.0.0.1:36582
I20260319 16:33:21.788955 31055 tablet_replica.cc:333] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: stopping tablet replica
I20260319 16:33:21.789037 31055 raft_consensus.cc:2243] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:21.789121 31056 tablet_replica.cc:333] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: stopping tablet replica
I20260319 16:33:21.789131 31055 raft_consensus.cc:2272] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:21.789247 31056 raft_consensus.cc:2243] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:21.789330 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 30627
I20260319 16:33:21.789520 31055 ts_tablet_manager.cc:1916] T 02ad4a66fa10454fabc3da9a304cc756 P 951f1a3d70a94c0d9122b4cea7814007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:21.791162 31056 raft_consensus.cc:2272] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:21.791584 31056 ts_tablet_manager.cc:1916] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:21.793201 31056 ts_tablet_manager.cc:1929] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:21.793270 31056 log.cc:1199] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/02ad4a66fa10454fabc3da9a304cc756
I20260319 16:33:21.793525 31056 ts_tablet_manager.cc:1950] T 02ad4a66fa10454fabc3da9a304cc756 P 9fb8491cb29f4155b31fc7f35e2d4bf3: Deleting consensus metadata
I20260319 16:33:21.793906 30552 catalog_manager.cc:5002] TS 9fb8491cb29f4155b31fc7f35e2d4bf3 (127.24.187.194:32883): tablet 02ad4a66fa10454fabc3da9a304cc756 (table test-table [id=5c05d2859e204aa198a1da8a49a54cf4]) successfully deleted
W20260319 16:33:21.795336 30554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:45979 (error 108)
I20260319 16:33:21.795796 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.795671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:45979 (local address 127.0.0.1:56104)
0319 16:33:21.795723 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:21.795762 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260319 16:33:21.795853 31054 tablet_replica.cc:333] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: stopping tablet replica
I20260319 16:33:21.795846 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 30758
I20260319 16:33:21.795935 31054 raft_consensus.cc:2243] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:21.796031 31054 raft_consensus.cc:2272] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:21.797148 31054 ts_tablet_manager.cc:1916] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260319 16:33:21.798159 30554 catalog_manager.cc:4729] TS 951f1a3d70a94c0d9122b4cea7814007 (127.24.187.193:45979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02ad4a66fa10454fabc3da9a304cc756: Network error: Client connection negotiation failed: client connection to 127.24.187.193:45979: connect: Connection refused (error 111)
I20260319 16:33:21.798835 31054 ts_tablet_manager.cc:1929] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:21.798894 31054 log.cc:1199] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/02ad4a66fa10454fabc3da9a304cc756
I20260319 16:33:21.799085 31054 ts_tablet_manager.cc:1950] T 02ad4a66fa10454fabc3da9a304cc756 P b8ec04da1a74436ca07c0fe771321da3: Deleting consensus metadata
I20260319 16:33:21.799485 30553 catalog_manager.cc:5002] TS b8ec04da1a74436ca07c0fe771321da3 (127.24.187.195:37203): tablet 02ad4a66fa10454fabc3da9a304cc756 (table test-table [id=5c05d2859e204aa198a1da8a49a54cf4]) successfully deleted
I20260319 16:33:21.801461 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 30889
I20260319 16:33:21.805770 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 30536
2026-03-19T16:33:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (644 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260319 16:33:21.824350 25327 mini_oidc.cc:150] Starting JWKS server
I20260319 16:33:21.825304 25327 webserver.cc:492] Webserver started at https://127.0.0.1:39865/ using document root <none> and password file <none>
I20260319 16:33:21.825376 25327 mini_oidc.cc:186] Starting OIDC Discovery server
I20260319 16:33:21.825505 25327 webserver.cc:492] Webserver started at http://127.0.0.1:36407/ using document root <none> and password file <none>
2026-03-19T16:33:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:21Z Disabled control of system clock
I20260319 16:33:21.832257 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38451
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:38451
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/testchainca.cert with env {}
W20260319 16:33:21.904592 31063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:21.904769 31063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:21.904798 31063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:21.906248 31063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:21.906291 31063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:21.906306 31063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:21.906322 31063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:21.906337 31063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260319 16:33:21.906352 31063 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260319 16:33:21.906373 31063 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39865/jwks.json
I20260319 16:33:21.907999 31063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:38451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:21.908231 31063 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:21.908383 31063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:21.910656 31071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:21.910662 31069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.910833 31063 server_base.cc:1060] running on GCE node
W20260319 16:33:21.910707 31068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:21.911115 31063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:21.911304 31063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:21.912433 31063 hybrid_clock.cc:648] HybridClock initialized: now 1773938001912410 us; error 34 us; skew 500 ppm
I20260319 16:33:21.913439 31063 webserver.cc:492] Webserver started at http://127.24.187.254:37251/ using document root <none> and password file <none>
I20260319 16:33:21.913614 31063 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:21.913666 31063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:21.913787 31063 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:21.914695 31063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "ffab97cf41e94e4f8ed054c4beba75b8"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "8efe22c30dbaf61aa1d87d7a027fe0e5"
server_key_iv: "7a2fdb0b1bf1b0b4f661076891a7af72"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.914978 31063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "ffab97cf41e94e4f8ed054c4beba75b8"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "8efe22c30dbaf61aa1d87d7a027fe0e5"
server_key_iv: "7a2fdb0b1bf1b0b4f661076891a7af72"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.916085 31063 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:21.916816 31077 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:21.917003 31063 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:21.917073 31063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "ffab97cf41e94e4f8ed054c4beba75b8"
format_stamp: "Formatted at 2026-03-19 16:33:21 on dist-test-slave-svn8"
server_key: "8efe22c30dbaf61aa1d87d7a027fe0e5"
server_key_iv: "7a2fdb0b1bf1b0b4f661076891a7af72"
server_key_version: "encryptionkey@0"
I20260319 16:33:21.917137 31063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:21.932022 31063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:21.932263 31063 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:21.932365 31063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:21.936008 31063 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:38451
I20260319 16:33:21.936041 31129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:38451 every 8 connection(s)
I20260319 16:33:21.936316 31063 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:21.936816 31130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:21.939108 31130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8: Bootstrap starting.
I20260319 16:33:21.939667 31130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:21.939882 31130 log.cc:826] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:21.940488 31130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8: No bootstrap required, opened a new log
I20260319 16:33:21.941648 31130 raft_consensus.cc:359] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38451 } }
I20260319 16:33:21.941809 31130 raft_consensus.cc:385] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:21.941844 31130 raft_consensus.cc:740] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffab97cf41e94e4f8ed054c4beba75b8, State: Initialized, Role: FOLLOWER
I20260319 16:33:21.941959 31130 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38451 } }
I20260319 16:33:21.942009 31130 raft_consensus.cc:399] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:21.942039 31130 raft_consensus.cc:493] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:21.942078 31130 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:21.942601 31130 raft_consensus.cc:515] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38451 } }
I20260319 16:33:21.942695 31130 leader_election.cc:304] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffab97cf41e94e4f8ed054c4beba75b8; no voters:
I20260319 16:33:21.942833 31130 leader_election.cc:290] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:21.942881 31133 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:21.943002 31133 raft_consensus.cc:697] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [term 1 LEADER]: Becoming Leader. State: Replica: ffab97cf41e94e4f8ed054c4beba75b8, State: Running, Role: LEADER
I20260319 16:33:21.943043 31130 sys_catalog.cc:565] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:21.943081 31133 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38451 } }
I20260319 16:33:21.943385 31134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38451 } } }
I20260319 16:33:21.943444 31135 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffab97cf41e94e4f8ed054c4beba75b8. Latest consensus state: current_term: 1 leader_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffab97cf41e94e4f8ed054c4beba75b8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38451 } } }
I20260319 16:33:21.943543 31135 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:21.943717 31139 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:21.944183 31139 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:21.943732 31134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:21.945806 31139 catalog_manager.cc:1357] Generated new cluster ID: 87fd9433fd6e4e2ca9ab8f44664c6bda
I20260319 16:33:21.945856 31139 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:21.945909 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31063
I20260319 16:33:21.946007 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:21.946187 25327 external_mini_cluster.cc:1468] Setting key a4d408e92790dc308bf257502855cacf
I20260319 16:33:21.954062 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.946575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.24.187.254:38451)
0319 16:33:21.946750 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.946757 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.949947 (+ 3190us) server_negotiation.cc:408] Connection header received
0319 16:33:21.950072 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.950078 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.950149 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.950283 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:21.950688 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.951913 (+ 1225us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.952707 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.952962 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.953310 (+ 348us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:21.953317 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:21.953317 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:21.953565 (+ 248us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:21.953595 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:21.953604 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:21.953685 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:21.953836 (+ 151us) server_negotiation.cc:300] Negotiation successful
0319 16:33:21.953897 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260319 16:33:21.955055 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:33:21.962090 31139 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:21.962809 31139 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:21.967737 31139 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ffab97cf41e94e4f8ed054c4beba75b8: Generated new TSK 0
I20260319 16:33:21.967892 31139 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:33:22.003579 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:21.977371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.24.187.254:38451)
0319 16:33:21.977416 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:21.977418 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:21.977539 (+ 121us) server_negotiation.cc:408] Connection header received
0319 16:33:21.977637 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:21.977638 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:21.977648 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:21.977698 (+ 50us) server_negotiation.cc:227] Negotiated authn=JWT
0319 16:33:21.978112 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.978649 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:21.979470 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:21.979597 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:21.979631 (+ 34us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0319 16:33:22.003218 (+ 23587us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0319 16:33:22.003279 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:22.003449 (+ 170us) server_negotiation.cc:300] Negotiation successful
0319 16:33:22.003489 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24,"thread_start_us":105,"threads_started":1}
I20260319 16:33:22.011718 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:22.009062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.24.187.254:38451)
0319 16:33:22.009105 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:22.009108 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:22.009158 (+ 50us) server_negotiation.cc:408] Connection header received
0319 16:33:22.009240 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:22.009241 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:22.009249 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:22.009306 (+ 57us) server_negotiation.cc:227] Negotiated authn=JWT
0319 16:33:22.009639 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:22.010147 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:22.010954 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:22.011085 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:22.011102 (+ 17us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0319 16:33:22.011566 (+ 464us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0319 16:33:22.011671 (+ 105us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57652: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":23}
W20260319 16:33:22.011778 31153 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57652: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260319 16:33:22.011994 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:22.008961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38451 (local address 127.0.0.1:57652)
0319 16:33:22.009089 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:22.009100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:22.009166 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:22.009322 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:22.009323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:22.009325 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0319 16:33:22.009548 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:22.009551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:22.010236 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:22.010237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:22.010860 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:22.010862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:22.010941 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:22.010947 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0319 16:33:22.011674 (+ 727us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0319 16:33:22.011776 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.187.254:38451: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":81,"threads_started":1}
I20260319 16:33:37.018447 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:37.016287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.24.187.254:38451)
0319 16:33:37.016488 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.016490 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.016507 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:33:37.016533 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.016534 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.016544 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.016596 (+ 52us) server_negotiation.cc:227] Negotiated authn=JWT
0319 16:33:37.016858 (+ 262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.017355 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.017944 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.018080 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.018098 (+ 18us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0319 16:33:37.018264 (+ 166us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0319 16:33:37.018370 (+ 106us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34964: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
W20260319 16:33:37.018491 31171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:37.016190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38451 (local address 127.0.0.1:34964)
0319 16:33:37.016359 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:37.016370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:37.016433 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:37.016609 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:37.016610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:37.016612 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0319 16:33:37.016788 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.016790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.017427 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.017428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:37.017870 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.017872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.017936 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.017942 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0319 16:33:37.018334 (+ 392us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0319 16:33:37.018430 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.187.254:38451: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
W20260319 16:33:37.018522 31172 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34964: JWT verification failed: token expired
W20260319 16:33:37.019865 31172 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:37.019516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.24.187.254:38451)
0319 16:33:37.019559 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.019561 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.019621 (+ 60us) server_negotiation.cc:408] Connection header received
0319 16:33:37.019665 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.019665 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.019675 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.019735 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:37.019820 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34974: BlockingRecv error: recv got EOF from 127.0.0.1:34974 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260319 16:33:37.022401 31182 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.24.187.254:38451
W20260319 16:33:37.022720 31172 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:37.022255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.24.187.254:38451)
0319 16:33:37.022303 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.022305 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.022390 (+ 85us) server_negotiation.cc:408] Connection header received
0319 16:33:37.022505 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.022506 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.022514 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.022565 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:37.022652 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34988: BlockingRecv error: recv got EOF from 127.0.0.1:34988 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260319 16:33:37.022811 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31063
2026-03-19T16:33:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260319 16:33:37.060930 25327 mini_oidc.cc:150] Starting JWKS server
I20260319 16:33:37.061687 25327 webserver.cc:492] Webserver started at https://127.0.0.1:42045/ using document root <none> and password file <none>
I20260319 16:33:37.061766 25327 mini_oidc.cc:186] Starting OIDC Discovery server
I20260319 16:33:37.061904 25327 webserver.cc:492] Webserver started at http://127.0.0.1:40413/ using document root <none> and password file <none>
2026-03-19T16:33:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:37Z Disabled control of system clock
I20260319 16:33:37.069104 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39903
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:36101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39903
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42045/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/testchainca.cert with env {}
W20260319 16:33:37.142766 31188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:37.142951 31188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:37.143081 31188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:37.144838 31188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:37.144892 31188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:37.144903 31188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:37.144914 31188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:37.144924 31188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260319 16:33:37.144935 31188 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260319 16:33:37.144944 31188 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42045/jwks.json
I20260319 16:33:37.146770 31188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:36101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42045/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:37.147123 31188 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:37.147328 31188 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:33:37.150027 31188 server_base.cc:1060] running on GCE node
W20260319 16:33:37.150116 31196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:37.150210 31193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:37.150197 31194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:37.150550 31188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:37.150779 31188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:37.151916 31188 hybrid_clock.cc:648] HybridClock initialized: now 1773938017151897 us; error 29 us; skew 500 ppm
I20260319 16:33:37.152995 31188 webserver.cc:492] Webserver started at http://127.24.187.254:34789/ using document root <none> and password file <none>
I20260319 16:33:37.153168 31188 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:37.153218 31188 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:37.153319 31188 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:37.154333 31188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "b3a90fff759744b49d4dd77baeb592f5"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "de434bb3b9b3b458c74d9146ff864634"
server_key_iv: "ddb6c52f11f7164ec152956c65aa1280"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.154670 31188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "b3a90fff759744b49d4dd77baeb592f5"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "de434bb3b9b3b458c74d9146ff864634"
server_key_iv: "ddb6c52f11f7164ec152956c65aa1280"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.155782 31188 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:37.156571 31202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:37.156765 31188 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:37.156826 31188 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "b3a90fff759744b49d4dd77baeb592f5"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "de434bb3b9b3b458c74d9146ff864634"
server_key_iv: "ddb6c52f11f7164ec152956c65aa1280"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.156872 31188 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:37.168061 31188 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:37.168326 31188 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:37.168483 31188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:37.172806 31188 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39903
I20260319 16:33:37.172909 31254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39903 every 8 connection(s)
I20260319 16:33:37.173157 31188 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:37.173823 31255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:37.176398 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5: Bootstrap starting.
I20260319 16:33:37.176947 31255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:37.177136 31255 log.cc:826] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:37.177801 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5: No bootstrap required, opened a new log
I20260319 16:33:37.178968 31255 raft_consensus.cc:359] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a90fff759744b49d4dd77baeb592f5" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39903 } }
I20260319 16:33:37.179086 31255 raft_consensus.cc:385] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:37.179105 31255 raft_consensus.cc:740] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a90fff759744b49d4dd77baeb592f5, State: Initialized, Role: FOLLOWER
I20260319 16:33:37.179198 31255 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a90fff759744b49d4dd77baeb592f5" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39903 } }
I20260319 16:33:37.179257 31255 raft_consensus.cc:399] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:37.179292 31255 raft_consensus.cc:493] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:37.179322 31255 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:37.179813 31255 raft_consensus.cc:515] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a90fff759744b49d4dd77baeb592f5" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39903 } }
I20260319 16:33:37.179898 31255 leader_election.cc:304] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3a90fff759744b49d4dd77baeb592f5; no voters:
I20260319 16:33:37.180009 31255 leader_election.cc:290] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:37.180084 31258 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:37.180193 31258 raft_consensus.cc:697] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [term 1 LEADER]: Becoming Leader. State: Replica: b3a90fff759744b49d4dd77baeb592f5, State: Running, Role: LEADER
I20260319 16:33:37.180249 31255 sys_catalog.cc:565] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:37.180266 31258 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a90fff759744b49d4dd77baeb592f5" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39903 } }
I20260319 16:33:37.180604 31259 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3a90fff759744b49d4dd77baeb592f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a90fff759744b49d4dd77baeb592f5" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39903 } } }
I20260319 16:33:37.180634 31260 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3a90fff759744b49d4dd77baeb592f5. Latest consensus state: current_term: 1 leader_uuid: "b3a90fff759744b49d4dd77baeb592f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a90fff759744b49d4dd77baeb592f5" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39903 } } }
I20260319 16:33:37.180733 31260 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:37.180927 31264 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:37.181406 31264 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:37.180938 31259 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:37.182837 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31188
I20260319 16:33:37.182919 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:37.182982 31264 catalog_manager.cc:1357] Generated new cluster ID: a9da35346011474d95723566396954fa
I20260319 16:33:37.183023 31264 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:37.183063 25327 external_mini_cluster.cc:1468] Setting key f469619993999e72ed67bb6cd5ac6c1e
I20260319 16:33:37.190235 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:37.183488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.24.187.254:39903)
0319 16:33:37.183679 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.183686 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.186019 (+ 2333us) server_negotiation.cc:408] Connection header received
0319 16:33:37.186140 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.186147 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.186203 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.186332 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:37.186757 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.187967 (+ 1210us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.188834 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.189112 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.189341 (+ 229us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:37.189347 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:37.189348 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:37.189587 (+ 239us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:37.189632 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:37.189642 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:37.189749 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:37.189810 (+ 61us) server_negotiation.cc:300] Negotiation successful
0319 16:33:37.190055 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260319 16:33:37.191244 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:33:37.199981 31264 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:37.200697 31264 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:37.205797 31264 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b3a90fff759744b49d4dd77baeb592f5: Generated new TSK 0
I20260319 16:33:37.205983 31264 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260319 16:33:37.230643 31285 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260319 16:33:37.230758 31285 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260319 16:33:37.231055 31284 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:37.215955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39903 (local address 127.0.0.1:42420)
0319 16:33:37.216091 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:37.216101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:37.216166 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:37.216314 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:37.216315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:37.216317 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0319 16:33:37.216512 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.216514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.217487 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.217488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:37.217964 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.217967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.218035 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.218040 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0319 16:33:37.230853 (+ 12813us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42045/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0319 16:33:37.230949 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.187.254:39903: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42045/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260319 16:33:37.231462 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:37.216052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.24.187.254:39903)
0319 16:33:37.216100 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.216102 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.216164 (+ 62us) server_negotiation.cc:408] Connection header received
0319 16:33:37.216227 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.216228 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.216240 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.216312 (+ 72us) server_negotiation.cc:227] Negotiated authn=JWT
0319 16:33:37.216567 (+ 255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.217383 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.218047 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.218236 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.218291 (+ 55us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0319 16:33:37.230721 (+ 12430us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42045/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0319 16:33:37.231276 (+ 555us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42045/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:33:37.231492 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31188
W20260319 16:33:37.231530 31278 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42045/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
2026-03-19T16:33:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (207 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260319 16:33:37.268613 25327 mini_oidc.cc:150] Starting JWKS server
I20260319 16:33:37.269340 25327 webserver.cc:492] Webserver started at https://127.0.0.1:46265/ using document root <none> and password file <none>
I20260319 16:33:37.269387 25327 mini_oidc.cc:186] Starting OIDC Discovery server
I20260319 16:33:37.269506 25327 webserver.cc:492] Webserver started at http://127.0.0.1:44959/ using document root <none> and password file <none>
2026-03-19T16:33:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:37Z Disabled control of system clock
I20260319 16:33:37.276466 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39455
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39455
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46265/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260319 16:33:37.349885 31291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:37.350054 31291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:37.350080 31291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:37.351537 31291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:37.351583 31291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:37.351599 31291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:37.351615 31291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:37.351629 31291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260319 16:33:37.351651 31291 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260319 16:33:37.351672 31291 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46265/jwks.json
I20260319 16:33:37.353178 31291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46265/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:37.353423 31291 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:37.353623 31291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:37.356865 31297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:37.356882 31291 server_base.cc:1060] running on GCE node
W20260319 16:33:37.356865 31299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:37.356866 31296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:37.357206 31291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:37.357465 31291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:37.358623 31291 hybrid_clock.cc:648] HybridClock initialized: now 1773938017358601 us; error 31 us; skew 500 ppm
I20260319 16:33:37.359815 31291 webserver.cc:492] Webserver started at http://127.24.187.254:39911/ using document root <none> and password file <none>
I20260319 16:33:37.359994 31291 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:37.360034 31291 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:37.360139 31291 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:37.361065 31291 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "1c73203e1b814ecebf8c3f2e81c4d148"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "e62f274ecfa2f52efe19264a029449ce"
server_key_iv: "76f0f099aa5ed52a80e2b2bcf2595521"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.361351 31291 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "1c73203e1b814ecebf8c3f2e81c4d148"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "e62f274ecfa2f52efe19264a029449ce"
server_key_iv: "76f0f099aa5ed52a80e2b2bcf2595521"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.362651 31291 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:33:37.363322 31305 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:37.363570 31291 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:37.363637 31291 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "1c73203e1b814ecebf8c3f2e81c4d148"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "e62f274ecfa2f52efe19264a029449ce"
server_key_iv: "76f0f099aa5ed52a80e2b2bcf2595521"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.363706 31291 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:37.374821 31291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:37.375080 31291 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:37.375188 31291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:37.379096 31291 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39455
I20260319 16:33:37.379163 31357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39455 every 8 connection(s)
I20260319 16:33:37.379447 31291 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:37.380034 31358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:37.380079 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31291
I20260319 16:33:37.380162 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:37.380296 25327 external_mini_cluster.cc:1468] Setting key cc050d64e588df04d4330c6028be63e4
I20260319 16:33:37.382855 31358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148: Bootstrap starting.
I20260319 16:33:37.383386 31358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:37.383663 31358 log.cc:826] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:37.384255 31358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148: No bootstrap required, opened a new log
I20260319 16:33:37.384378 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:37.380787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.24.187.254:39455)
0319 16:33:37.380979 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.380984 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.381000 (+ 16us) server_negotiation.cc:408] Connection header received
0319 16:33:37.381327 (+ 327us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.381332 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.381377 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.381480 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:37.381920 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.382626 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.383234 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.383519 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.383876 (+ 357us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:37.383884 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:37.383884 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:37.384027 (+ 143us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:37.384053 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:37.384060 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:37.384117 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:37.384181 (+ 64us) server_negotiation.cc:300] Negotiation successful
0319 16:33:37.384235 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260319 16:33:37.385267 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:33:37.385545 31358 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39455 } }
I20260319 16:33:37.385667 31358 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:37.385687 31358 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c73203e1b814ecebf8c3f2e81c4d148, State: Initialized, Role: FOLLOWER
I20260319 16:33:37.385807 31358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39455 } }
I20260319 16:33:37.385864 31358 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:37.385890 31358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:37.385934 31358 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:37.386515 31358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39455 } }
I20260319 16:33:37.386626 31358 leader_election.cc:304] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c73203e1b814ecebf8c3f2e81c4d148; no voters:
I20260319 16:33:37.386759 31358 leader_election.cc:290] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:37.386817 31363 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:37.386921 31363 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [term 1 LEADER]: Becoming Leader. State: Replica: 1c73203e1b814ecebf8c3f2e81c4d148, State: Running, Role: LEADER
I20260319 16:33:37.386981 31358 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:37.386996 31363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39455 } }
I20260319 16:33:37.387326 31364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39455 } } }
I20260319 16:33:37.387352 31365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c73203e1b814ecebf8c3f2e81c4d148. Latest consensus state: current_term: 1 leader_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c73203e1b814ecebf8c3f2e81c4d148" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39455 } } }
I20260319 16:33:37.387445 31365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:37.387406 31364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:37.387817 31369 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:37.388257 31369 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:37.389679 31369 catalog_manager.cc:1357] Generated new cluster ID: 9efd9b248539434a99efb90a9d3fa07d
I20260319 16:33:37.389755 31369 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:37.415524 31369 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:37.416256 31369 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:37.425814 31369 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1c73203e1b814ecebf8c3f2e81c4d148: Generated new TSK 0
I20260319 16:33:37.426011 31369 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260319 16:33:37.454934 31388 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260319 16:33:37.455013 31388 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260319 16:33:37.456229 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:37.437252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.24.187.254:39455)
0319 16:33:37.437300 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.437302 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.437374 (+ 72us) server_negotiation.cc:408] Connection header received
0319 16:33:37.437446 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.437447 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.437460 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.437533 (+ 73us) server_negotiation.cc:227] Negotiated authn=JWT
0319 16:33:37.437873 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.438666 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.439252 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.439435 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.439485 (+ 50us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0319 16:33:37.455946 (+ 16461us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0319 16:33:37.456156 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260319 16:33:37.456295 31361 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260319 16:33:37.456287 31387 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:37.437166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39455 (local address 127.0.0.1:34678)
0319 16:33:37.437295 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:37.437306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:37.437373 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:37.437567 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:37.437568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:37.437571 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0319 16:33:37.437795 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.437798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.438759 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.438760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:37.439176 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.439178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.439252 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.439257 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0319 16:33:37.456081 (+ 16824us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0319 16:33:37.456193 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.187.254:39455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260319 16:33:37.456604 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31291
2026-03-19T16:33:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-03-19T16:33:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:37Z Disabled control of system clock
I20260319 16:33:37.500706 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42759
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:42759
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/testchainca.cert with env {}
W20260319 16:33:37.574774 31392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:37.574940 31392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:37.574971 31392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:37.576424 31392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:37.576467 31392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:37.576483 31392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:37.576499 31392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:37.576514 31392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260319 16:33:37.576529 31392 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260319 16:33:37.576548 31392 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260319 16:33:37.578171 31392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:42759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:37.578373 31392 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:37.578528 31392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:37.580888 31398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:37.580866 31400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:37.580881 31397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:37.581041 31392 server_base.cc:1060] running on GCE node
I20260319 16:33:37.581317 31392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:37.581550 31392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:37.582693 31392 hybrid_clock.cc:648] HybridClock initialized: now 1773938017582660 us; error 42 us; skew 500 ppm
I20260319 16:33:37.583743 31392 webserver.cc:492] Webserver started at http://127.24.187.254:33883/ using document root <none> and password file <none>
I20260319 16:33:37.583909 31392 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:37.583948 31392 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:37.584048 31392 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:37.584928 31392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "e80cc4eaee5845dbba63d4bab6641ea1"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "ca07871ef890fe007744da9de03fecda"
server_key_iv: "822ba03d8be5b19215416ceee76dd173"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.585211 31392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "e80cc4eaee5845dbba63d4bab6641ea1"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "ca07871ef890fe007744da9de03fecda"
server_key_iv: "822ba03d8be5b19215416ceee76dd173"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.586347 31392 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:37.587121 31406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:37.587296 31392 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:37.587359 31392 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "e80cc4eaee5845dbba63d4bab6641ea1"
format_stamp: "Formatted at 2026-03-19 16:33:37 on dist-test-slave-svn8"
server_key: "ca07871ef890fe007744da9de03fecda"
server_key_iv: "822ba03d8be5b19215416ceee76dd173"
server_key_version: "encryptionkey@0"
I20260319 16:33:37.587425 31392 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:37.615670 31392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:37.615955 31392 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:37.616047 31392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:37.619721 31392 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:42759
I20260319 16:33:37.619756 31458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:42759 every 8 connection(s)
I20260319 16:33:37.620034 31392 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:37.620584 31459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:37.622879 31459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1: Bootstrap starting.
I20260319 16:33:37.623407 31459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:37.623620 31459 log.cc:826] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:37.624248 31459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1: No bootstrap required, opened a new log
I20260319 16:33:37.624495 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31392
I20260319 16:33:37.624579 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:37.624722 25327 external_mini_cluster.cc:1468] Setting key e02dad34d2bad42a5d6ef0b7ca15c6f0
I20260319 16:33:37.625603 31459 raft_consensus.cc:359] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42759 } }
I20260319 16:33:37.625768 31459 raft_consensus.cc:385] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:37.625809 31459 raft_consensus.cc:740] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80cc4eaee5845dbba63d4bab6641ea1, State: Initialized, Role: FOLLOWER
I20260319 16:33:37.625895 31459 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42759 } }
I20260319 16:33:37.625953 31459 raft_consensus.cc:399] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:37.625981 31459 raft_consensus.cc:493] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:37.626024 31459 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:37.626554 31459 raft_consensus.cc:515] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42759 } }
I20260319 16:33:37.626665 31459 leader_election.cc:304] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e80cc4eaee5845dbba63d4bab6641ea1; no voters:
I20260319 16:33:37.626791 31459 leader_election.cc:290] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:37.626835 31464 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:37.626943 31464 raft_consensus.cc:697] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [term 1 LEADER]: Becoming Leader. State: Replica: e80cc4eaee5845dbba63d4bab6641ea1, State: Running, Role: LEADER
I20260319 16:33:37.627013 31459 sys_catalog.cc:565] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:37.627017 31464 consensus_queue.cc:237] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42759 } }
I20260319 16:33:37.627300 31464 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80cc4eaee5845dbba63d4bab6641ea1. Latest consensus state: current_term: 1 leader_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42759 } } }
I20260319 16:33:37.627360 31464 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:37.627550 31468 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:37.627535 31465 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e80cc4eaee5845dbba63d4bab6641ea1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42759 } } }
I20260319 16:33:37.628304 31465 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:37.629070 31468 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:37.629142 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:37.625155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.24.187.254:42759)
0319 16:33:37.625350 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.625355 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.625370 (+ 15us) server_negotiation.cc:408] Connection header received
0319 16:33:37.625567 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.625571 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.625612 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.625741 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:37.626138 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.626970 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.627786 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.628035 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.628411 (+ 376us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:37.628418 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:37.628419 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:33:37.628624 (+ 205us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:37.628660 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:33:37.628669 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:37.628735 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:37.628922 (+ 187us) server_negotiation.cc:300] Negotiation successful
0319 16:33:37.628977 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260319 16:33:37.630234 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:33:37.630932 31468 catalog_manager.cc:1357] Generated new cluster ID: f303e5e0ce394c68a107b99cd7ccf1e4
I20260319 16:33:37.630986 31468 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:37.640746 31468 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:37.641448 31468 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:37.648702 31468 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e80cc4eaee5845dbba63d4bab6641ea1: Generated new TSK 0
I20260319 16:33:37.648874 31468 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:33:37.742415 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:37.661684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.24.187.254:42759)
0319 16:33:37.661757 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:37.661760 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:37.661843 (+ 83us) server_negotiation.cc:408] Connection header received
0319 16:33:37.661939 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:37.661940 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:37.661952 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:37.662001 (+ 49us) server_negotiation.cc:227] Negotiated authn=JWT
0319 16:33:37.662354 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.662886 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.663702 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.663847 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.663889 (+ 42us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0319 16:33:37.742191 (+ 78302us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0319 16:33:37.742343 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260319 16:33:37.742478 31463 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260319 16:33:37.742586 31488 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:37.661580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42759 (local address 127.0.0.1:52486)
0319 16:33:37.661769 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:37.661780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:37.661852 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:37.662013 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:37.662014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:37.662016 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0319 16:33:37.662253 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.662255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.662966 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:37.662967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:37.663609 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:37.663612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:37.663701 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:37.663707 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0319 16:33:37.742322 (+ 78615us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0319 16:33:37.742470 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.187.254:42759: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260319 16:33:37.742901 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31392
2026-03-19T16:33:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (266 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:37 dist-test-slave-svn8 krb5kdc[31500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:37 dist-test-slave-svn8 krb5kdc[31500](info): set up 2 sockets
Mar 19 16:33:37 dist-test-slave-svn8 krb5kdc[31500](info): commencing operation
krb5kdc: starting...
W20260319 16:33:39.766805 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.831s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:39 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:39.900187 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:32997
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:39.974602 31516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:39.974774 31516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:39.974804 31516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:39.976271 31516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:39.976317 31516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:39.976334 31516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:39.976358 31516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:39.976374 31516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:39.978062 31516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:39.978281 31516 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:39.978444 31516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:39.980777 31524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:39.980772 31521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:39.980777 31522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:39.980959 31516 server_base.cc:1060] running on GCE node
I20260319 16:33:39.981132 31516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:39.981364 31516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:39.982581 31516 hybrid_clock.cc:648] HybridClock initialized: now 1773938019982542 us; error 30 us; skew 500 ppm
Mar 19 16:33:39 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938019, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:39.985077 31516 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:39.985627 31516 webserver.cc:492] Webserver started at http://127.24.187.254:40651/ using document root <none> and password file <none>
I20260319 16:33:39.985831 31516 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:39.985867 31516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:39.985971 31516 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:39.986893 31516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "1764fc72edc24315b90189e718707593"
format_stamp: "Formatted at 2026-03-19 16:33:39 on dist-test-slave-svn8"
server_key: "f4d020f4794b8daa9809b660641c31f8"
server_key_iv: "d46467d599fa97014ccb289f76e96c16"
server_key_version: "encryptionkey@0"
I20260319 16:33:39.987185 31516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "1764fc72edc24315b90189e718707593"
format_stamp: "Formatted at 2026-03-19 16:33:39 on dist-test-slave-svn8"
server_key: "f4d020f4794b8daa9809b660641c31f8"
server_key_iv: "d46467d599fa97014ccb289f76e96c16"
server_key_version: "encryptionkey@0"
I20260319 16:33:39.988294 31516 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:39.989055 31531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:39.989243 31516 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:39.989303 31516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "1764fc72edc24315b90189e718707593"
format_stamp: "Formatted at 2026-03-19 16:33:39 on dist-test-slave-svn8"
server_key: "f4d020f4794b8daa9809b660641c31f8"
server_key_iv: "d46467d599fa97014ccb289f76e96c16"
server_key_version: "encryptionkey@0"
I20260319 16:33:39.989405 31516 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:40.003273 31516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:40.008785 31516 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:40.008917 31516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:40.012794 31516 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:32997
I20260319 16:33:40.012820 31583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:32997 every 8 connection(s)
I20260319 16:33:40.013118 31516 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:40.013681 31584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:40.014577 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31516
I20260319 16:33:40.014676 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:40.014817 25327 external_mini_cluster.cc:1468] Setting key defa0ade5361a780b2239c4a4e361bd2
I20260319 16:33:40.016491 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593: Bootstrap starting.
I20260319 16:33:40.017096 31584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:40.017336 31584 log.cc:826] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:40.017982 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593: No bootstrap required, opened a new log
I20260319 16:33:40.019286 31584 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764fc72edc24315b90189e718707593" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 32997 } }
I20260319 16:33:40.019418 31584 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:40.019438 31584 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1764fc72edc24315b90189e718707593, State: Initialized, Role: FOLLOWER
I20260319 16:33:40.019534 31584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764fc72edc24315b90189e718707593" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 32997 } }
I20260319 16:33:40.019594 31584 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:40.019615 31584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:40.019656 31584 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 0 FOLLOWER]: Advancing to term 1
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:40.020188 31584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764fc72edc24315b90189e718707593" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 32997 } }
I20260319 16:33:40.020279 31584 leader_election.cc:304] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1764fc72edc24315b90189e718707593; no voters:
I20260319 16:33:40.020422 31584 leader_election.cc:290] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:40.020462 31589 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:40.020555 31589 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [term 1 LEADER]: Becoming Leader. State: Replica: 1764fc72edc24315b90189e718707593, State: Running, Role: LEADER
I20260319 16:33:40.020648 31584 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:40.020625 31589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764fc72edc24315b90189e718707593" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 32997 } }
I20260319 16:33:40.020902 31589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1764fc72edc24315b90189e718707593. Latest consensus state: current_term: 1 leader_uuid: "1764fc72edc24315b90189e718707593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764fc72edc24315b90189e718707593" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 32997 } } }
I20260319 16:33:40.020975 31589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:40.021026 31590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1764fc72edc24315b90189e718707593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1764fc72edc24315b90189e718707593" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 32997 } } }
I20260319 16:33:40.021096 31590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:40.021210 31593 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:40.021826 31593 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:40.023967 31593 catalog_manager.cc:1357] Generated new cluster ID: 0b7885836abe4462b9264e02d8a34c32
I20260319 16:33:40.024014 31593 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:40.024292 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.015288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.24.187.254:32997)
0319 16:33:40.015447 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.015458 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.015480 (+ 22us) server_negotiation.cc:408] Connection header received
0319 16:33:40.015875 (+ 395us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.015883 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.015935 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.016066 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.016760 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.017788 (+ 1028us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.018323 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.018600 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.020507 (+ 1907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:40.020515 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:40.020515 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:40.020536 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:40.023042 (+ 2506us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.023336 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.023339 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.023340 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.023383 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.023543 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.023544 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.023544 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.023768 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:40.023908 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:40.024023 (+ 115us) server_negotiation.cc:300] Negotiation successful
0319 16:33:40.024134 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260319 16:33:40.044189 31593 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:40.044819 31593 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:40.048033 31593 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1764fc72edc24315b90189e718707593: Generated new TSK 0
I20260319 16:33:40.048194 31593 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:40.104311 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:32997
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:40.178927 31612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:40.179085 31612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:40.179105 31612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:40.180539 31612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:40.180580 31612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:40.180604 31612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:40.182216 31612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:40.182397 31612 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:40.182559 31612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:40.184947 31620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:40.184952 31617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:40.184952 31618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:40.185122 31612 server_base.cc:1060] running on GCE node
I20260319 16:33:40.185333 31612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:40.185555 31612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:40.186681 31612 hybrid_clock.cc:648] HybridClock initialized: now 1773938020186649 us; error 42 us; skew 500 ppm
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.188895 31612 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:40.189399 31612 webserver.cc:492] Webserver started at http://127.24.187.193:35735/ using document root <none> and password file <none>
I20260319 16:33:40.189566 31612 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:40.189600 31612 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:40.189741 31612 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:40.190696 31612 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "f218d98305e7498c8c228b97858c0f02"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "5c2d6045a3a40accde6fbc4ef3ddf18e"
server_key_iv: "b097d224b9983eec8289e8f07a2437c4"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.190982 31612 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "f218d98305e7498c8c228b97858c0f02"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "5c2d6045a3a40accde6fbc4ef3ddf18e"
server_key_iv: "b097d224b9983eec8289e8f07a2437c4"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.192095 31612 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:40.192832 31627 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.192996 31612 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:40.193056 31612 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "f218d98305e7498c8c228b97858c0f02"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "5c2d6045a3a40accde6fbc4ef3ddf18e"
server_key_iv: "b097d224b9983eec8289e8f07a2437c4"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.193118 31612 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:40.210935 31612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:40.212759 31612 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:40.212885 31612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:40.213078 31612 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:40.213409 31612 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:40.213449 31612 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.213479 31612 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:40.213497 31612 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.219058 31612 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:35805
I20260319 16:33:40.219102 31740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:35805 every 8 connection(s)
I20260319 16:33:40.219377 31612 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:40.227216 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.220018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:52843 (local address 127.24.187.254:32997)
0319 16:33:40.220060 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.220063 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.220221 (+ 158us) server_negotiation.cc:408] Connection header received
0319 16:33:40.220730 (+ 509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.220731 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.220739 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.220800 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.221549 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.222129 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.222697 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.222813 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.224972 (+ 2159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:40.224977 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:40.224978 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:40.224986 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:40.226259 (+ 1273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.226602 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.226603 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.226603 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.226628 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.226768 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.226769 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.226769 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.226852 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:40.226894 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:40.227080 (+ 186us) server_negotiation.cc:300] Negotiation successful
0319 16:33:40.227150 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:33:40.227303 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.219986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:32997 (local address 127.24.187.193:52843)
0319 16:33:40.220130 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:40.220162 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:40.220652 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:40.220847 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:40.220850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:40.221136 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:40.221461 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:40.221467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.222202 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.222203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:40.222624 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:40.222626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.222720 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.223526 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:40.223540 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:40.224866 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:40.226323 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:40.226327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:40.226329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:40.226538 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:40.226677 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:40.226677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:40.226677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:40.226728 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:40.226891 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:40.226893 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:40.226989 (+ 96us) client_negotiation.cc:769] Sending connection context
0319 16:33:40.227066 (+ 77us) client_negotiation.cc:240] Negotiation successful
0319 16:33:40.227162 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260319 16:33:40.227763 31741 heartbeater.cc:344] Connected to a master server at 127.24.187.254:32997
I20260319 16:33:40.227847 31741 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:40.228022 31741 heartbeater.cc:507] Master 127.24.187.254:32997 requested a full tablet report, sending...
I20260319 16:33:40.228489 31548 ts_manager.cc:194] Registered new tserver with Master: f218d98305e7498c8c228b97858c0f02 (127.24.187.193:35805)
I20260319 16:33:40.229090 31548 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:52843
I20260319 16:33:40.229384 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31612
I20260319 16:33:40.229490 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:40.229648 25327 external_mini_cluster.cc:1468] Setting key 76074a6f898e20e6f4459664d9f7dba4
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:40.274492 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:32997
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:40.349289 31748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:40.349458 31748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:40.349480 31748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:40.350910 31748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:40.350952 31748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:40.350977 31748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:40.352512 31748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:40.352696 31748 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:40.352856 31748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:40.355222 31756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:40.355312 31753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:40.355327 31754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:40.355436 31748 server_base.cc:1060] running on GCE node
I20260319 16:33:40.355597 31748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:40.355783 31748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:40.356942 31748 hybrid_clock.cc:648] HybridClock initialized: now 1773938020356920 us; error 33 us; skew 500 ppm
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.359287 31748 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:40.359807 31748 webserver.cc:492] Webserver started at http://127.24.187.194:35059/ using document root <none> and password file <none>
I20260319 16:33:40.359965 31748 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:40.359999 31748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:40.360096 31748 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:40.360957 31748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "d825161e2b0a4491b40da6a297214d32"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "9687deaa3befae1d3ebf4ffce273de85"
server_key_iv: "15784ee394b43c4678a48eb166ec8fa8"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.361240 31748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "d825161e2b0a4491b40da6a297214d32"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "9687deaa3befae1d3ebf4ffce273de85"
server_key_iv: "15784ee394b43c4678a48eb166ec8fa8"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.362387 31748 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:40.363049 31763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.363214 31748 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:40.363272 31748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "d825161e2b0a4491b40da6a297214d32"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "9687deaa3befae1d3ebf4ffce273de85"
server_key_iv: "15784ee394b43c4678a48eb166ec8fa8"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.363332 31748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:40.382927 31748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:40.384620 31748 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:40.384740 31748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:40.384933 31748 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:40.385206 31748 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:40.385237 31748 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.385267 31748 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:40.385290 31748 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.390888 31748 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:35481
I20260319 16:33:40.390939 31876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:35481 every 8 connection(s)
I20260319 16:33:40.391336 31748 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:40.399077 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31748
I20260319 16:33:40.399174 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:40.399366 25327 external_mini_cluster.cc:1468] Setting key bcadf48011c58437149565d6c859f4af
I20260319 16:33:40.399485 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.391988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:51683 (local address 127.24.187.254:32997)
0319 16:33:40.392069 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.392071 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.392210 (+ 139us) server_negotiation.cc:408] Connection header received
0319 16:33:40.392754 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.392756 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.392766 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.392825 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.393609 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.394152 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.394732 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.394859 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.397083 (+ 2224us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:40.397087 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:40.397088 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:40.397094 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:40.398281 (+ 1187us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.398630 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.398631 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.398631 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.398649 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.398842 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.398843 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.398843 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.398949 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:40.399000 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:40.399322 (+ 322us) server_negotiation.cc:300] Negotiation successful
0319 16:33:40.399397 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260319 16:33:40.399639 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.391936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:32997 (local address 127.24.187.194:51683)
0319 16:33:40.392077 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:40.392108 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:40.392657 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:40.392872 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:40.392875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:40.393213 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:40.393525 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:40.393529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.394246 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.394247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:40.394656 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:40.394658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.394754 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.395634 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:40.395649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:40.397007 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:40.398369 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:40.398374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:40.398376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:40.398569 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:40.398715 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:40.398716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:40.398717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:40.398784 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:40.399032 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:40.399038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:40.399182 (+ 144us) client_negotiation.cc:769] Sending connection context
0319 16:33:40.399305 (+ 123us) client_negotiation.cc:240] Negotiation successful
0319 16:33:40.399441 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260319 16:33:40.400205 31877 heartbeater.cc:344] Connected to a master server at 127.24.187.254:32997
I20260319 16:33:40.400295 31877 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:40.400513 31877 heartbeater.cc:507] Master 127.24.187.254:32997 requested a full tablet report, sending...
I20260319 16:33:40.400882 31547 ts_manager.cc:194] Registered new tserver with Master: d825161e2b0a4491b40da6a297214d32 (127.24.187.194:35481)
I20260319 16:33:40.401345 31547 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:51683
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:40.444128 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:32997
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:40.517014 31884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:40.517171 31884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:40.517199 31884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:40.518679 31884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:40.518725 31884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:40.518756 31884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:40.520296 31884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:40.520493 31884 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:40.520658 31884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:40.523102 31889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:40.523082 31890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:40.523097 31892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:40.523418 31884 server_base.cc:1060] running on GCE node
I20260319 16:33:40.523571 31884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:40.523759 31884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:40.524906 31884 hybrid_clock.cc:648] HybridClock initialized: now 1773938020524888 us; error 32 us; skew 500 ppm
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.527231 31884 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:40.527725 31884 webserver.cc:492] Webserver started at http://127.24.187.195:40067/ using document root <none> and password file <none>
I20260319 16:33:40.527884 31884 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:40.527917 31884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:40.528013 31884 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:40.528904 31884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "2854391234264c2485b1d48846811ddb"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "a88e3c7bd13784a17912bf3c42f16958"
server_key_iv: "63b102bfbe47b7edf1631872cc428299"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.529189 31884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "2854391234264c2485b1d48846811ddb"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "a88e3c7bd13784a17912bf3c42f16958"
server_key_iv: "63b102bfbe47b7edf1631872cc428299"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.530344 31884 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:33:40.530990 31899 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.531167 31884 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:40.531234 31884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "2854391234264c2485b1d48846811ddb"
format_stamp: "Formatted at 2026-03-19 16:33:40 on dist-test-slave-svn8"
server_key: "a88e3c7bd13784a17912bf3c42f16958"
server_key_iv: "63b102bfbe47b7edf1631872cc428299"
server_key_version: "encryptionkey@0"
I20260319 16:33:40.531297 31884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:40.550961 31884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:40.552771 31884 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:40.552892 31884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:40.553081 31884 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:40.553401 31884 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:40.553439 31884 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.553467 31884 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:40.553493 31884 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:40.559024 31884 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:40919
I20260319 16:33:40.559072 32012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:40919 every 8 connection(s)
I20260319 16:33:40.559381 31884 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:40.567178 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.560038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:47395 (local address 127.24.187.254:32997)
0319 16:33:40.560085 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.560087 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.560225 (+ 138us) server_negotiation.cc:408] Connection header received
0319 16:33:40.560746 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.560747 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.560756 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.560814 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.561530 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.562047 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.562682 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.562827 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.564915 (+ 2088us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:40.564919 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:40.564920 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:40.564927 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:40.566100 (+ 1173us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.566440 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.566441 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.566441 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.566458 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.566693 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.566693 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.566694 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.566776 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:40.566826 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:40.567050 (+ 224us) server_negotiation.cc:300] Negotiation successful
0319 16:33:40.567113 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260319 16:33:40.567268 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.559995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:32997 (local address 127.24.187.195:47395)
0319 16:33:40.560134 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:40.560165 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:40.560674 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:40.560888 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:40.560891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:40.561168 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:40.561460 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:40.561463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.562148 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.562149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:40.562575 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:40.562577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.562680 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.563507 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:40.563519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:40.564833 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:40.566171 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:40.566175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:40.566177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:40.566381 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:40.566531 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:40.566532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:40.566532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:40.566598 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:40.566824 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:40.566827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:40.566940 (+ 113us) client_negotiation.cc:769] Sending connection context
0319 16:33:40.567032 (+ 92us) client_negotiation.cc:240] Negotiation successful
0319 16:33:40.567132 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260319 16:33:40.567667 32013 heartbeater.cc:344] Connected to a master server at 127.24.187.254:32997
I20260319 16:33:40.567737 32013 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:40.567901 32013 heartbeater.cc:507] Master 127.24.187.254:32997 requested a full tablet report, sending...
I20260319 16:33:40.568243 31547 ts_manager.cc:194] Registered new tserver with Master: 2854391234264c2485b1d48846811ddb (127.24.187.195:40919)
I20260319 16:33:40.568571 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 31884
I20260319 16:33:40.568651 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:40.568717 31547 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:47395
I20260319 16:33:40.568801 25327 external_mini_cluster.cc:1468] Setting key 82a41651fb1dae8b5338951668db4372
I20260319 16:33:40.569666 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/test-admin.keytab.
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.584018 25327 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260319 16:33:40.585460 32020 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:40.584817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.24.187.193:35805)
0319 16:33:40.584984 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.584990 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.585000 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:40.585026 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.585028 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.585042 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.585101 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.585386 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51424: BlockingRecv error: recv got EOF from 127.0.0.1:51424 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260319 16:33:40.585567 25327 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5cc' of type FILE
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.586535 25327 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.587378 25327 init.cc:295] Successfully reacquired a new kerberos TGT
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:33:40.594158 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.587935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.24.187.193:35805)
0319 16:33:40.587981 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.587983 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.587992 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:40.588060 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.588061 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.588079 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.588129 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.588642 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.589331 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.589916 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.590118 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.591826 (+ 1708us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:40.591850 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:40.591851 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:40.591858 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:40.593280 (+ 1422us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.593581 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.593583 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.593584 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.593612 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.593796 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.593797 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.593797 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.593883 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:40.593994 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:40.594035 (+ 41us) server_negotiation.cc:300] Negotiation successful
0319 16:33:40.594096 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260319 16:33:40.595889 32020 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:40.595404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.24.187.193:35805)
0319 16:33:40.595454 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.595455 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.595487 (+ 32us) server_negotiation.cc:408] Connection header received
0319 16:33:40.595523 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.595524 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.595532 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.595573 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.595853 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51442: BlockingRecv error: recv got EOF from 127.0.0.1:51442 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260319 16:33:40.595973 25327 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5cc' of type FILE
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.596864 25327 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.597672 25327 init.cc:295] Successfully reacquired a new kerberos TGT
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:33:40.603820 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.598223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.24.187.193:35805)
0319 16:33:40.598267 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.598269 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.598292 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:33:40.598465 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.598466 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.598477 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.598527 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.599014 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.599479 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.599989 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.600156 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.601800 (+ 1644us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:40.601804 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:40.601804 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:40.601810 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:40.602883 (+ 1073us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.603142 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.603143 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.603143 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.603162 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.603270 (+ 108us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.603271 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.603271 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.603335 (+ 64us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:40.603399 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:40.603646 (+ 247us) server_negotiation.cc:300] Negotiation successful
0319 16:33:40.603728 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260319 16:33:40.605312 32020 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:40.604816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.24.187.193:35805)
0319 16:33:40.604867 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.604869 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.604915 (+ 46us) server_negotiation.cc:408] Connection header received
0319 16:33:40.604972 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.604973 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.604982 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.605029 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.605277 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51460: BlockingRecv error: recv got EOF from 127.0.0.1:51460 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260319 16:33:40.605458 25327 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1773937967898507-25327-0/krb5kdc/krb5cc' of type FILE
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.606395 25327 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:40.607259 25327 init.cc:295] Successfully reacquired a new kerberos TGT
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[31500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:33:40.612959 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:40.607766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.24.187.193:35805)
0319 16:33:40.607809 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:40.607810 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:40.607820 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:40.607873 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:40.607873 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:40.607881 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:40.607927 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:40.608340 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.608774 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:40.609297 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:40.609452 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:40.611090 (+ 1638us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:40.611092 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:40.611092 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:40.611098 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:40.612249 (+ 1151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.612511 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.612512 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.612512 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.612526 (+ 14us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:40.612645 (+ 119us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:40.612645 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:40.612646 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:40.612703 (+ 57us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:40.612759 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:40.612829 (+ 70us) server_negotiation.cc:300] Negotiation successful
0319 16:33:40.612884 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:40.613543 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31612
I20260319 16:33:40.618832 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31748
I20260319 16:33:40.623049 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31884
I20260319 16:33:40.627463 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 31516
2026-03-19T16:33:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2711 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[32033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[32033](info): set up 2 sockets
Mar 19 16:33:40 dist-test-slave-svn8 krb5kdc[32033](info): commencing operation
krb5kdc: starting...
W20260319 16:33:42.462875 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:42 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:42.598088 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:45081
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:45081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:42.673601 32049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:42.673810 32049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:42.673840 32049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:42.675304 32049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:42.675354 32049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:42.675374 32049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:42.675388 32049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:42.675403 32049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:42.676975 32049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:45081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:45081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:42.677186 32049 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:42.677341 32049 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:42.679594 32057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:42.679600 32055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:42.679602 32054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:42.680086 32049 server_base.cc:1060] running on GCE node
I20260319 16:33:42.680300 32049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:42.680573 32049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:42.681749 32049 hybrid_clock.cc:648] HybridClock initialized: now 1773938022681693 us; error 30 us; skew 500 ppm
Mar 19 16:33:42 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:42.683976 32049 init.cc:377] Logged in from keytab as oryx/127.24.187.254@KRBTEST.COM (short username oryx)
I20260319 16:33:42.684465 32049 webserver.cc:492] Webserver started at http://127.24.187.254:34047/ using document root <none> and password file <none>
I20260319 16:33:42.684614 32049 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:42.684643 32049 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:42.684716 32049 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:42.685578 32049 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "41f56935045b4f47be93f73ad5537b88"
format_stamp: "Formatted at 2026-03-19 16:33:42 on dist-test-slave-svn8"
server_key: "545e9101460e6e3c294fe20d22d3377c"
server_key_iv: "c49e26ce4bdfc65b26f64b3d3dffd6ea"
server_key_version: "encryptionkey@0"
I20260319 16:33:42.685889 32049 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "41f56935045b4f47be93f73ad5537b88"
format_stamp: "Formatted at 2026-03-19 16:33:42 on dist-test-slave-svn8"
server_key: "545e9101460e6e3c294fe20d22d3377c"
server_key_iv: "c49e26ce4bdfc65b26f64b3d3dffd6ea"
server_key_version: "encryptionkey@0"
I20260319 16:33:42.686964 32049 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:42.687577 32064 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:42.687750 32049 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:42.687811 32049 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "41f56935045b4f47be93f73ad5537b88"
format_stamp: "Formatted at 2026-03-19 16:33:42 on dist-test-slave-svn8"
server_key: "545e9101460e6e3c294fe20d22d3377c"
server_key_iv: "c49e26ce4bdfc65b26f64b3d3dffd6ea"
server_key_version: "encryptionkey@0"
I20260319 16:33:42.687860 32049 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:42.697594 32049 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:42.699517 32049 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:42.699626 32049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:42.703362 32049 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:45081
I20260319 16:33:42.703442 32116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:45081 every 8 connection(s)
I20260319 16:33:42.703696 32049 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:42.704443 32117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:42.707016 32117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: Bootstrap starting.
I20260319 16:33:42.707587 32117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:42.707808 32117 log.cc:826] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:42.708581 32117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: No bootstrap required, opened a new log
I20260319 16:33:42.709817 32117 raft_consensus.cc:359] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } }
I20260319 16:33:42.709935 32117 raft_consensus.cc:385] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:42.709954 32117 raft_consensus.cc:740] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f56935045b4f47be93f73ad5537b88, State: Initialized, Role: FOLLOWER
I20260319 16:33:42.710057 32117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } }
I20260319 16:33:42.710119 32117 raft_consensus.cc:399] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:42.710140 32117 raft_consensus.cc:493] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:42.710165 32117 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:42.710700 32117 raft_consensus.cc:515] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } }
I20260319 16:33:42.710791 32117 leader_election.cc:304] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f56935045b4f47be93f73ad5537b88; no voters:
I20260319 16:33:42.710932 32117 leader_election.cc:290] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:42.710991 32120 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:42.711133 32117 sys_catalog.cc:565] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:42.711133 32120 raft_consensus.cc:697] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 1 LEADER]: Becoming Leader. State: Replica: 41f56935045b4f47be93f73ad5537b88, State: Running, Role: LEADER
I20260319 16:33:42.711252 32120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } }
I20260319 16:33:42.711707 32121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } }
I20260319 16:33:42.711723 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f56935045b4f47be93f73ad5537b88. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } }
I20260319 16:33:42.711827 32121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:42.711833 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:42.712031 32129 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:42.712365 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 32049
I20260319 16:33:42.712525 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:42.712705 25327 external_mini_cluster.cc:1468] Setting key 7e74bb2b6c2444160365c82708f91d56
I20260319 16:33:42.712713 32129 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:42.714458 32129 catalog_manager.cc:1357] Generated new cluster ID: a509a400332f44e79b72a654d1fa34e2
I20260319 16:33:42.714515 32129 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:33:42 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:42.722560 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:42.713167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.24.187.254:45081)
0319 16:33:42.713385 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:42.713403 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:42.713420 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:33:42.713499 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:42.713503 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:42.713538 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:42.713643 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:42.714806 (+ 1163us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:42.715979 (+ 1173us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:42.716693 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:42.716931 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:42.719227 (+ 2296us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:42.719233 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:42.719233 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:42.719253 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:42.721090 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:42.721437 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:42.721439 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:42.721440 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:42.721491 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:42.721631 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:42.721632 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:42.721632 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:42.721880 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:42.721991 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:42.722117 (+ 126us) server_negotiation.cc:300] Negotiation successful
0319 16:33:42.722238 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260319 16:33:42.723534 32129 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:42.724045 32129 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:42.729450 32129 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: Generated new TSK 0
I20260319 16:33:42.729599 32129 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:42.780046 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45081
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:42.855883 32145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:42.856052 32145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:42.856076 32145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:42.857540 32145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:42.857585 32145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:42.857610 32145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:42.859257 32145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:45081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:42.859460 32145 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:42.859628 32145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:42.861961 32150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:42.861987 32151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:42.862169 32145 server_base.cc:1060] running on GCE node
W20260319 16:33:42.862283 32153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:42.862469 32145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:42.862669 32145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:42.863833 32145 hybrid_clock.cc:648] HybridClock initialized: now 1773938022863814 us; error 31 us; skew 500 ppm
Mar 19 16:33:42 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:42.866281 32145 init.cc:377] Logged in from keytab as oryx/127.24.187.193@KRBTEST.COM (short username oryx)
I20260319 16:33:42.866830 32145 webserver.cc:492] Webserver started at http://127.24.187.193:46651/ using document root <none> and password file <none>
I20260319 16:33:42.867010 32145 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:42.867048 32145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:42.867151 32145 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:42.868042 32145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "595ada663e79499f80cb25638ccf6ebb"
format_stamp: "Formatted at 2026-03-19 16:33:42 on dist-test-slave-svn8"
server_key: "d2439b24bb75574942512c5bfd35a091"
server_key_iv: "3f22b179e4bf44941e12ad4631c3a2f0"
server_key_version: "encryptionkey@0"
I20260319 16:33:42.868335 32145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "595ada663e79499f80cb25638ccf6ebb"
format_stamp: "Formatted at 2026-03-19 16:33:42 on dist-test-slave-svn8"
server_key: "d2439b24bb75574942512c5bfd35a091"
server_key_iv: "3f22b179e4bf44941e12ad4631c3a2f0"
server_key_version: "encryptionkey@0"
I20260319 16:33:42.869458 32145 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:42.870143 32160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:42.870309 32145 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:42.870375 32145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "595ada663e79499f80cb25638ccf6ebb"
format_stamp: "Formatted at 2026-03-19 16:33:42 on dist-test-slave-svn8"
server_key: "d2439b24bb75574942512c5bfd35a091"
server_key_iv: "3f22b179e4bf44941e12ad4631c3a2f0"
server_key_version: "encryptionkey@0"
I20260319 16:33:42.870441 32145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:42.882517 32145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:42.883984 32145 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:42.884094 32145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:42.884281 32145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:42.884550 32145 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:42.884588 32145 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:42.884617 32145 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:42.884639 32145 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:42.890074 32145 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:36825
I20260319 16:33:42.890121 32273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:36825 every 8 connection(s)
I20260319 16:33:42.890383 32145 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:42.894392 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 32145
I20260319 16:33:42.894488 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:42.894649 25327 external_mini_cluster.cc:1468] Setting key f869b10e915f7d63687b0671d71f8abb
Mar 19 16:33:42 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:42.898665 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:42.891034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:37581 (local address 127.24.187.254:45081)
0319 16:33:42.891107 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:42.891109 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:42.891184 (+ 75us) server_negotiation.cc:408] Connection header received
0319 16:33:42.891700 (+ 516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:42.891701 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:42.891714 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:42.891777 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:42.892454 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:42.892945 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:42.893561 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:42.893696 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:42.896024 (+ 2328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:42.896028 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:42.896028 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:42.896036 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:42.897312 (+ 1276us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:42.897762 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:42.897763 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:42.897764 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:42.897794 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:42.898026 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:42.898027 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:42.898028 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:42.898123 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:42.898187 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:42.898478 (+ 291us) server_negotiation.cc:300] Negotiation successful
0319 16:33:42.898596 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260319 16:33:42.898818 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:42.890964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.193:37581)
0319 16:33:42.891094 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:42.891124 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:42.891628 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:42.891816 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:42.891819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:42.892083 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:42.892354 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:42.892357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:42.893040 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:42.893042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:42.893468 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:42.893470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:42.893568 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:42.894031 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:42.894046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:42.895942 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:42.897389 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:42.897395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:42.897397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:42.897632 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:42.897865 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:42.897866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:42.897866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:42.897940 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:42.898187 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:42.898192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:42.898338 (+ 146us) client_negotiation.cc:769] Sending connection context
0319 16:33:42.898454 (+ 116us) client_negotiation.cc:240] Negotiation successful
0319 16:33:42.898593 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260319 16:33:42.899263 32274 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45081
I20260319 16:33:42.899365 32274 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:42.899531 32274 heartbeater.cc:507] Master 127.24.187.254:45081 requested a full tablet report, sending...
I20260319 16:33:42.899979 32081 ts_manager.cc:194] Registered new tserver with Master: 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:42.900599 32081 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:37581
WARNING: no policy specified for oryx/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:42.939960 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45081
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:43.013343 32281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:43.013530 32281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:43.013553 32281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:43.015046 32281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:43.015090 32281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:43.015116 32281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:43.016741 32281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:45081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:43.016907 32281 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:43.017035 32281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:43.019408 32287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.019462 32289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.019543 32281 server_base.cc:1060] running on GCE node
W20260319 16:33:43.019626 32286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.019802 32281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:43.019997 32281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:43.021163 32281 hybrid_clock.cc:648] HybridClock initialized: now 1773938023021144 us; error 25 us; skew 500 ppm
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:43.023484 32281 init.cc:377] Logged in from keytab as oryx/127.24.187.194@KRBTEST.COM (short username oryx)
I20260319 16:33:43.023965 32281 webserver.cc:492] Webserver started at http://127.24.187.194:35605/ using document root <none> and password file <none>
I20260319 16:33:43.024123 32281 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.024158 32281 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.024257 32281 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:43.025153 32281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "58d9dd27bba64c15963825d9e53958dc"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "e923b5d6f6198a6070b1d90db311c1fe"
server_key_iv: "08ace35a5ec3497a3c446e5d9833d053"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.025449 32281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "58d9dd27bba64c15963825d9e53958dc"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "e923b5d6f6198a6070b1d90db311c1fe"
server_key_iv: "08ace35a5ec3497a3c446e5d9833d053"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.026551 32281 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:43.027179 32296 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.027388 32281 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:43.027457 32281 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "58d9dd27bba64c15963825d9e53958dc"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "e923b5d6f6198a6070b1d90db311c1fe"
server_key_iv: "08ace35a5ec3497a3c446e5d9833d053"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.027518 32281 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.035593 32281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:43.036926 32281 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:43.037035 32281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:43.037209 32281 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:43.037477 32281 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:43.037508 32281 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.037533 32281 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:43.037556 32281 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.043098 32281 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:46733
I20260319 16:33:43.043167 32409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:46733 every 8 connection(s)
I20260319 16:33:43.043408 32281 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:43.044207 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 32281
I20260319 16:33:43.044360 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:43.044529 25327 external_mini_cluster.cc:1468] Setting key c3099ffcdc33a04a5a9bf327993bebd4
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.187.195@KRBTEST.COM; defaulting to no policy
I20260319 16:33:43.052337 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.044089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:40021 (local address 127.24.187.254:45081)
0319 16:33:43.044170 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.044172 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.044249 (+ 77us) server_negotiation.cc:408] Connection header received
0319 16:33:43.044939 (+ 690us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.044940 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.044952 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.045026 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.045901 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.046568 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.047198 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.047315 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.049966 (+ 2651us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.049970 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.049970 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.049977 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.051278 (+ 1301us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.051626 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.051627 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.051627 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.051650 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.051800 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.051801 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.051801 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.051889 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.051943 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.052164 (+ 221us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.052248 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260319 16:33:43.052454 32412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.043987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:40021)
0319 16:33:43.044132 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.044163 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.044866 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.045066 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.045069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.045388 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.045756 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.045794 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.046664 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.046665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.047109 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.047112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.047230 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.047763 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.047785 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.049884 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.051354 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.051360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.051362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.051557 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.051699 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.051700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.051700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.051753 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.051938 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.051942 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.052050 (+ 108us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.052137 (+ 87us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.052268 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260319 16:33:43.053058 32410 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45081
I20260319 16:33:43.053149 32410 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:43.053375 32410 heartbeater.cc:507] Master 127.24.187.254:45081 requested a full tablet report, sending...
I20260319 16:33:43.053762 32081 ts_manager.cc:194] Registered new tserver with Master: 58d9dd27bba64c15963825d9e53958dc (127.24.187.194:46733)
I20260319 16:33:43.054164 32081 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:40021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:43.090243 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:45081
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:43.163175 32417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:43.163360 32417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:43.163384 32417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:43.164848 32417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:43.164892 32417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:43.164918 32417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:43.166518 32417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:45081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:43.166720 32417 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:43.166877 32417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:43.169179 32423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.169245 32422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.169277 32417 server_base.cc:1060] running on GCE node
W20260319 16:33:43.169179 32425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.169493 32417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:43.169685 32417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:43.170861 32417 hybrid_clock.cc:648] HybridClock initialized: now 1773938023170846 us; error 27 us; skew 500 ppm
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:43.173130 32417 init.cc:377] Logged in from keytab as oryx/127.24.187.195@KRBTEST.COM (short username oryx)
I20260319 16:33:43.173635 32417 webserver.cc:492] Webserver started at http://127.24.187.195:39085/ using document root <none> and password file <none>
I20260319 16:33:43.173815 32417 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.173875 32417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.173982 32417 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:43.174865 32417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "800bf9287be4a3df5a6b59561a556b0a"
server_key_iv: "0dc91fa5e83d0df36cae9b41b83b2d60"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.175114 32417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "800bf9287be4a3df5a6b59561a556b0a"
server_key_iv: "0dc91fa5e83d0df36cae9b41b83b2d60"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.176156 32417 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:43.176843 32432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.177048 32417 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:43.177110 32417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "800bf9287be4a3df5a6b59561a556b0a"
server_key_iv: "0dc91fa5e83d0df36cae9b41b83b2d60"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.177165 32417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.191897 32417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:43.193660 32417 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:43.193817 32417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:43.194016 32417 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:43.194314 32417 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:43.194347 32417 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.194368 32417 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:43.194379 32417 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.199741 32417 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:45293
I20260319 16:33:43.199820 32545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:45293 every 8 connection(s)
I20260319 16:33:43.200060 32417 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:33:43.204542 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 32417
I20260319 16:33:43.204653 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:43.204838 25327 external_mini_cluster.cc:1468] Setting key aa21d30251ce89f57041737c307f4120
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:43.208379 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.200807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:54977 (local address 127.24.187.254:45081)
0319 16:33:43.200873 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.200875 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.201087 (+ 212us) server_negotiation.cc:408] Connection header received
0319 16:33:43.201540 (+ 453us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.201542 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.201552 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.201614 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.202507 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.203014 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.203602 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.203733 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.206200 (+ 2467us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.206203 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.206204 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.206210 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.207342 (+ 1132us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.207682 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.207682 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.207683 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.207699 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.207823 (+ 124us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.207823 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.207823 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.207903 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.207947 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.208233 (+ 286us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.208312 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:33:43.208583 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.200760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.195:54977)
0319 16:33:43.200925 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.200957 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.201455 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.201778 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.201783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.202119 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.202406 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.202410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.203134 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.203136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.203519 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.203521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.203635 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.204090 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.204107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.206129 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.207416 (+ 1287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.207421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.207423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.207606 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.207741 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.207741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.207742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.207788 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.207983 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.207988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.208104 (+ 116us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.208205 (+ 101us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.208340 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260319 16:33:43.209116 32546 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45081
I20260319 16:33:43.209209 32546 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:43.209396 32546 heartbeater.cc:507] Master 127.24.187.254:45081 requested a full tablet report, sending...
I20260319 16:33:43.209820 32081 ts_manager.cc:194] Registered new tserver with Master: 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195:45293)
I20260319 16:33:43.210186 32081 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:54977
I20260319 16:33:43.216607 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.187.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:43.260746 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:35849
--webserver_interface=127.24.187.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:45081,127.24.187.253:35849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:43.334807 32553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:43.334977 32553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:43.334998 32553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:43.336499 32553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:43.336547 32553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:43.336560 32553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:43.336570 32553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:43.336581 32553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:43.338241 32553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:45081,127.24.187.253:35849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:35849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.187.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
W20260319 16:33:43.338330 32553 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.187.254:45081,127.24.187.253:35849'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260319 16:33:43.338451 32553 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:43.338626 32553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:33:43.341012 32553 server_base.cc:1060] running on GCE node
W20260319 16:33:43.341089 32561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.341143 32559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.341159 32558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.341367 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:43.341588 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:43.342733 32553 hybrid_clock.cc:648] HybridClock initialized: now 1773938023342705 us; error 42 us; skew 500 ppm
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:43.345192 32553 init.cc:377] Logged in from keytab as oryx/127.24.187.253@KRBTEST.COM (short username oryx)
I20260319 16:33:43.345738 32553 webserver.cc:492] Webserver started at http://127.24.187.253:36759/ using document root <none> and password file <none>
I20260319 16:33:43.345909 32553 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.345944 32553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.346042 32553 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:43.346944 32553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/instance:
uuid: "919c26ab3f984ce6a3546c6ce1131317"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "d827415619387084ef964842db4fc3b8"
server_key_iv: "b1da5bcec0560fa1a9decb42d23f1520"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.347235 32553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal/instance:
uuid: "919c26ab3f984ce6a3546c6ce1131317"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "d827415619387084ef964842db4fc3b8"
server_key_iv: "b1da5bcec0560fa1a9decb42d23f1520"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.348344 32553 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:43.348945 32568 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.349104 32553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:43.349162 32553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
uuid: "919c26ab3f984ce6a3546c6ce1131317"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "d827415619387084ef964842db4fc3b8"
server_key_iv: "b1da5bcec0560fa1a9decb42d23f1520"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.349221 32553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.358626 32553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:43.360538 32553 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:43.360674 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:43.364418 32553 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.253:35849
I20260319 16:33:43.364460 32620 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.253:35849 every 8 connection(s)
I20260319 16:33:43.364737 32553 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
I20260319 16:33:43.365190 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 32553
I20260319 16:33:43.365257 32621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:43.367799 32621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.187.253@KRBTEST.COM
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.253@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:43.374536 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.365778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.24.187.253:35849)
0319 16:33:43.365905 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.365913 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.365931 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:33:43.366282 (+ 351us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.366288 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.366327 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.366427 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.367457 (+ 1030us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.368264 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.368853 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.369077 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.371522 (+ 2445us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.371529 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.371530 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.371536 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.373237 (+ 1701us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.373566 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.373570 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.373570 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.373653 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.373820 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.373821 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.373821 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.374008 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.374132 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.374265 (+ 133us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.374372 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260319 16:33:43.375479 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.368371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.24.187.254:45081)
0319 16:33:43.368452 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.368454 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.368572 (+ 118us) server_negotiation.cc:408] Connection header received
0319 16:33:43.368631 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.368632 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.368643 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.368690 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.369319 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.370240 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.370834 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.371094 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.372954 (+ 1860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.372959 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.372959 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.372967 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.374407 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.374771 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.374772 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.374773 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.374797 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.374973 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.374975 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.374975 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.375095 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.375180 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.375311 (+ 131us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.375388 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:33:43.375502 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.368308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39822)
0319 16:33:43.368490 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.368513 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.368572 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.368706 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.368708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.368978 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.369214 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.369217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.370335 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.370336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.370735 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.370737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.370834 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.371342 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.371361 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.372864 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.374499 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.374502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.374504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.374701 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.374846 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.374846 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.374847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.374874 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.375167 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.375170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.375208 (+ 38us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.375270 (+ 62us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.375410 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260319 16:33:43.375985 32621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.253@KRBTEST.COM for oryx/127.24.187.253@KRBTEST.COM
I20260319 16:33:43.381206 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.376208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34682)
0319 16:33:43.376248 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.376258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.376296 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.376425 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.376426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.376542 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.376676 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.376678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.377198 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.377199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.377515 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.377516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.377570 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.377972 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.377977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.379106 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.380428 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.380429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.380429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.380544 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.380727 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.380728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.380728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.380754 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.380921 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.380921 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.380953 (+ 32us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.381016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.381105 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260319 16:33:43.381245 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.376258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.24.187.253:35849)
0319 16:33:43.376293 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.376294 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.376302 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:43.376345 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.376346 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.376354 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.376421 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.376737 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.377141 (+ 404us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.377572 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.377772 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.379195 (+ 1423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.379198 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.379198 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.379203 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.380358 (+ 1155us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.380596 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.380597 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.380597 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.380613 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.380801 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.380802 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.380802 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.380875 (+ 73us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.380925 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.381048 (+ 123us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.381155 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":21}
I20260319 16:33:43.382501 32621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Bootstrap starting.
I20260319 16:33:43.383002 32621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:43.383419 32621 log.cc:826] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:43.384064 32621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: No bootstrap required, opened a new log
I20260319 16:33:43.385133 32621 raft_consensus.cc:359] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } }
I20260319 16:33:43.385236 32621 raft_consensus.cc:385] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:43.385255 32621 raft_consensus.cc:740] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919c26ab3f984ce6a3546c6ce1131317, State: Initialized, Role: FOLLOWER
I20260319 16:33:43.385329 32621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } }
I20260319 16:33:43.385586 32621 sys_catalog.cc:565] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:43.385584 32627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } } }
I20260319 16:33:43.385692 32627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: FOLLOWER
W20260319 16:33:43.387178 32638 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:33:43.387233 32638 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:33:43.388634 32553 master_runner.cc:428] Detected that this master 919c26ab3f984ce6a3546c6ce1131317 is joining an existing cluster
I20260319 16:33:43.388691 32553 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.187.253:35849
I20260319 16:33:43.394490 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.389522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39832)
0319 16:33:43.389701 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.389731 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.389812 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.389991 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.389992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.390211 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.390412 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.390414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.391120 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.391121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.391470 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.391471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.391528 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.391984 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.391989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.392225 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.393654 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.393655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.393655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.393837 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.394010 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.394010 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.394011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.394034 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.394240 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.394241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.394285 (+ 44us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.394330 (+ 45us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.394404 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":133,"threads_started":1}
I20260319 16:33:43.394551 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.389573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.24.187.254:45081)
0319 16:33:43.389624 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.389626 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.389786 (+ 160us) server_negotiation.cc:408] Connection header received
0319 16:33:43.389885 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.389887 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.389897 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.389925 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.390485 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.391023 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.391534 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.391694 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.392302 (+ 608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.392305 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.392305 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.392312 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.393578 (+ 1266us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.393909 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.393911 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.393911 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.393942 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.394076 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.394077 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.394078 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.394189 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.394254 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.394368 (+ 114us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.394457 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.254@KRBTEST.COM for oryx/127.24.187.253@KRBTEST.COM
I20260319 16:33:43.420836 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.396723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34708)
0319 16:33:43.396866 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.396901 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.396970 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.397144 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.397147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.397536 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.397840 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.397843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.398499 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.398500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.399063 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.399066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.399161 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.399638 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.399643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.401185 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.419649 (+ 18464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.419663 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.419666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.419985 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.420190 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.420192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.420193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.420237 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.420473 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.420483 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.420553 (+ 70us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.420651 (+ 98us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.420754 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260319 16:33:43.421409 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.396793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.24.187.253:35849)
0319 16:33:43.396934 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.396935 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.396946 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:43.397061 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.397063 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.397072 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.397131 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.397935 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.398430 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.399183 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.399362 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.401269 (+ 1907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.401272 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.401273 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.401279 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.419540 (+ 18261us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.420078 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.420079 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.420080 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.420109 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.420302 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.420303 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.420303 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.420409 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.420475 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.420887 (+ 412us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.420972 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260319 16:33:43.422008 32081 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.24.187.253:35849: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317"
member_type: NON_VOTER
last_known_addr {
host: "127.24.187.253"
port: 35849
}
attrs {
promote: true
}
}
dest_uuid: "41f56935045b4f47be93f73ad5537b88"
cas_config_opid_index: -1
I20260319 16:33:43.422308 32081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } }
I20260319 16:33:43.422802 32648 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 919c26ab3f984ce6a3546c6ce1131317 (127.24.187.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } }
I20260319 16:33:43.423080 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.389739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.24.187.253:35849)
0319 16:33:43.389774 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.389776 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.389864 (+ 88us) server_negotiation.cc:408] Connection header received
0319 16:33:43.389906 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.389907 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.389916 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.389970 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.418527 (+ 28557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.419229 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.419894 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.420046 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.420665 (+ 619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.420668 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.420668 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.420674 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.422198 (+ 1524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.422481 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.422482 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.422482 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.422508 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.422647 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.422647 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.422648 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.422734 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.422773 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.422930 (+ 157us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.423007 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:33:43.423098 32595 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:43.423373 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.389527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34698)
0319 16:33:43.389800 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.389808 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.389860 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.389988 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.389989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.418003 (+ 28014us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.418246 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.418252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.419334 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.419335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.419824 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.419826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.419889 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.420311 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.420316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.420595 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.422278 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.422279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.422279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.422413 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.422563 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.422564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.422564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.422586 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.422809 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.422810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.422842 (+ 32us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.423210 (+ 368us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.423294 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
I20260319 16:33:43.424027 32595 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Refusing update from remote peer 41f56935045b4f47be93f73ad5537b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260319 16:33:43.424190 32648 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.24.187.253:35849
I20260319 16:33:43.424422 32553 master.cc:561] Master@127.24.187.253:35849 shutting down...
I20260319 16:33:43.424396 32650 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:43.424649 32649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } }
I20260319 16:33:43.424715 32649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
W20260319 16:33:43.425446 32067 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260319 16:33:43.425917 32067 consensus_peers.cc:597] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 -> Peer 919c26ab3f984ce6a3546c6ce1131317 (127.24.187.253:35849): Couldn't send request to peer 919c26ab3f984ce6a3546c6ce1131317. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260319 16:33:43.427038 32553 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:43.427122 32553 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:43.427165 32553 tablet_replica.cc:333] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: stopping tablet replica
I20260319 16:33:43.439458 32553 master.cc:583] Master@127.24.187.253:35849 shutdown complete.
I20260319 16:33:43.439508 32553 master_runner.cc:305] Clearing existing system tablet
I20260319 16:33:43.439633 32553 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.439677 32553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.440228 32553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:43.440687 32654 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.440836 32553 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.440888 32553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.440909 32553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
uuid: "919c26ab3f984ce6a3546c6ce1131317"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "d827415619387084ef964842db4fc3b8"
server_key_iv: "b1da5bcec0560fa1a9decb42d23f1520"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.441488 32553 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:43.442966 32553 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260319 16:33:43.443043 32553 log.cc:1199] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260319 16:33:43.443219 32553 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Deleting consensus metadata
I20260319 16:33:43.443322 32553 master_runner.cc:315] Copying system tablet from 127.24.187.254:45081
I20260319 16:33:43.443944 32553 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: tablet copy: Beginning tablet copy session from remote peer at address 127.24.187.254:45081
I20260319 16:33:43.449602 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.444457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.24.187.254:45081)
0319 16:33:43.444518 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.444520 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.444531 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:43.444604 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.444605 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.444624 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.444670 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.445116 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.445610 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.446197 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.446368 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.447321 (+ 953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.447325 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.447326 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.447333 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.448706 (+ 1373us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.449009 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.449011 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.449011 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.449032 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.449157 (+ 125us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.449158 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.449159 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.449246 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.449300 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.449430 (+ 130us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.449507 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260319 16:33:43.449626 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.444254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39846)
0319 16:33:43.444425 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.444439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.444515 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.444696 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.444697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.444864 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.445033 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.445035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.445690 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.445691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.446116 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.446118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.446190 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.446755 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.446759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.447226 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.448798 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.448799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.448800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.448942 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.449080 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.449081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.449081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.449110 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.449294 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.449295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.449341 (+ 46us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.449401 (+ 60us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.449507 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260319 16:33:43.450223 32101 tablet_copy_service.cc:140] P 41f56935045b4f47be93f73ad5537b88: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 919c26ab3f984ce6a3546c6ce1131317 ({username='oryx', principal='oryx/127.24.187.253@KRBTEST.COM'} at 127.0.0.1:39846)
I20260319 16:33:43.450302 32101 tablet_copy_service.cc:161] P 41f56935045b4f47be93f73ad5537b88: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 919c26ab3f984ce6a3546c6ce1131317 at {username='oryx', principal='oryx/127.24.187.253@KRBTEST.COM'} at 127.0.0.1:39846: session id = 919c26ab3f984ce6a3546c6ce1131317-00000000000000000000000000000000
I20260319 16:33:43.450937 32101 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: Tablet Copy: opened 0 blocks and 1 log segments
I20260319 16:33:43.451728 32553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260319 16:33:43.452724 32553 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: tablet copy: Starting download of 0 data blocks...
I20260319 16:33:43.452847 32553 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: tablet copy: Starting download of 1 WAL segments...
I20260319 16:33:43.453758 32553 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260319 16:33:43.454665 32101 tablet_copy_service.cc:342] P 41f56935045b4f47be93f73ad5537b88: Request end of tablet copy session 919c26ab3f984ce6a3546c6ce1131317-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.187.253@KRBTEST.COM'} at 127.0.0.1:39846
I20260319 16:33:43.454732 32101 tablet_copy_service.cc:434] P 41f56935045b4f47be93f73ad5537b88: ending tablet copy session 919c26ab3f984ce6a3546c6ce1131317-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 919c26ab3f984ce6a3546c6ce1131317
W20260319 16:33:43.456435 32553 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260319 16:33:43.457432 32553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:43.458956 32662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.459038 32664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.458994 32661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.459326 32553 server_base.cc:1060] running on GCE node
I20260319 16:33:43.459623 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:43.459761 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:43.460883 32553 hybrid_clock.cc:648] HybridClock initialized: now 1773938023460876 us; error 26 us; skew 500 ppm
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:43.462363 32553 init.cc:377] Logged in from keytab as oryx/127.24.187.253@KRBTEST.COM (short username oryx)
I20260319 16:33:43.462604 32553 webserver.cc:492] Webserver started at http://127.24.187.253:37585/ using document root <none> and password file <none>
I20260319 16:33:43.462733 32553 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.462765 32553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.463367 32553 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:43.463837 32671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.464018 32553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.464077 32553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
uuid: "919c26ab3f984ce6a3546c6ce1131317"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "d827415619387084ef964842db4fc3b8"
server_key_iv: "b1da5bcec0560fa1a9decb42d23f1520"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.464143 32553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.479025 32553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:43.480366 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:43.483196 32553 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.253:35849
I20260319 16:33:43.483359 32723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.253:35849 every 8 connection(s)
I20260319 16:33:43.483533 32553 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
I20260319 16:33:43.484184 32724 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:33:43.484527 32724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Bootstrap starting.
I20260319 16:33:43.486490 32724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:33:43.486655 32724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Bootstrap complete.
I20260319 16:33:43.486860 32724 raft_consensus.cc:359] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } }
I20260319 16:33:43.486923 32724 raft_consensus.cc:740] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 919c26ab3f984ce6a3546c6ce1131317, State: Initialized, Role: LEARNER
I20260319 16:33:43.486946 32724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } }
I20260319 16:33:43.487090 32724 sys_catalog.cc:565] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:43.487108 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } }
I20260319 16:33:43.487203 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: LEARNER
I20260319 16:33:43.488423 32738 catalog_manager.cc:1269] Loaded cluster ID: a509a400332f44e79b72a654d1fa34e2
I20260319 16:33:43.488463 32738 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: loading cluster ID for follower catalog manager: success
I20260319 16:33:43.489637 32738 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: acquiring CA information for follower catalog manager: success
I20260319 16:33:43.489884 32738 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260319 16:33:43.535368 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.529429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.24.187.253:35849)
0319 16:33:43.529546 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.529547 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.529556 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:43.529581 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.529582 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.529590 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.529661 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.530533 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.531229 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.531964 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.532082 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.533142 (+ 1060us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.533165 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.533165 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.533171 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.534547 (+ 1376us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.534855 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.534855 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.534856 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.534873 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.534986 (+ 113us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.534987 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.534987 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.535066 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.535133 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.535225 (+ 92us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.535286 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for oryx/127.24.187.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:43.585238 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:42251
--webserver_interface=127.24.187.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:43.668260 32744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:43.668443 32744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:43.668473 32744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:43.670030 32744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:43.670073 32744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:43.670091 32744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:43.670106 32744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:43.670133 32744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:43.671734 32744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.187.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:43.671921 32744 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:43.672093 32744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:43.674727 32750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.674724 32749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.674722 32752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.675014 32744 server_base.cc:1060] running on GCE node
I20260319 16:33:43.675195 32744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:43.675400 32744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:43.676546 32744 hybrid_clock.cc:648] HybridClock initialized: now 1773938023676527 us; error 33 us; skew 500 ppm
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:43.679036 32744 init.cc:377] Logged in from keytab as oryx/127.24.187.252@KRBTEST.COM (short username oryx)
I20260319 16:33:43.679636 32744 webserver.cc:492] Webserver started at http://127.24.187.252:39481/ using document root <none> and password file <none>
I20260319 16:33:43.679805 32744 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.679847 32744 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.679944 32744 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:43.680976 32744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/instance:
uuid: "8fc58ffbf41b424e8e84eb71d38bd07c"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "24461b1d74c8160ccd356fe492d9d426"
server_key_iv: "a68c545954e599ed6b5e8b3a68ab6eb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.681277 32744 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal/instance:
uuid: "8fc58ffbf41b424e8e84eb71d38bd07c"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "24461b1d74c8160ccd356fe492d9d426"
server_key_iv: "a68c545954e599ed6b5e8b3a68ab6eb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.682492 32744 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:33:43.683173 32759 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.683367 32744 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:43.683454 32744 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
uuid: "8fc58ffbf41b424e8e84eb71d38bd07c"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "24461b1d74c8160ccd356fe492d9d426"
server_key_iv: "a68c545954e599ed6b5e8b3a68ab6eb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.683508 32744 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.695560 32744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:43.697415 32744 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:43.697558 32744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:43.701522 32744 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.252:42251
I20260319 16:33:43.701637 343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.252:42251 every 8 connection(s)
I20260319 16:33:43.701892 32744 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
I20260319 16:33:43.702479 344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:43.704226 344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.252@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:43.709903 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 32744
I20260319 16:33:43.712245 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.704977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.24.187.254:45081)
0319 16:33:43.705026 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.705028 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.705170 (+ 142us) server_negotiation.cc:408] Connection header received
0319 16:33:43.705503 (+ 333us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.705505 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.705514 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.705565 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.706311 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.706805 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.707334 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.707500 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.709210 (+ 1710us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.709215 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.709215 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.709222 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.710715 (+ 1493us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.711343 (+ 628us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.711344 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.711345 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.711369 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.711615 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.711617 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.711617 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.711752 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.711816 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.712032 (+ 216us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.712157 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:33:43.712255 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.704864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39854)
0319 16:33:43.705063 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.705088 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.705417 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.705641 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.705644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.705971 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.706228 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.706231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.706891 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.706892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.707263 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.707264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.707364 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.707812 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.707827 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.709126 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.710882 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.710886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.710889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.711250 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.711433 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.711433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.711434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.711530 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.711816 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.711819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.711919 (+ 100us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.711995 (+ 76us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.712098 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260319 16:33:43.712740 344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.187.252@KRBTEST.COM
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.252@KRBTEST.COM for oryx/127.24.187.253@KRBTEST.COM
I20260319 16:33:43.718386 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.710351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.24.187.252:42251)
0319 16:33:43.710510 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.710518 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.710537 (+ 19us) server_negotiation.cc:408] Connection header received
0319 16:33:43.710559 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.710561 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.710574 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.710652 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.711529 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.712651 (+ 1122us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.713291 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.713497 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.715689 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.715696 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.715696 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.715702 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.717408 (+ 1706us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.717684 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.717688 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.717688 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.717741 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.717877 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.717878 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.717878 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.717974 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.718102 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.718166 (+ 64us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.718275 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260319 16:33:43.719204 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.712999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34748)
0319 16:33:43.713113 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.713123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.713169 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.713315 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.713316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.713451 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.713624 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.713626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.714587 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.714588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.714948 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.714949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.715015 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.715402 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.715405 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.716577 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.718169 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.718170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.718171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.718408 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.718718 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.718718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.718719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.718747 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.718919 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.718919 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.718967 (+ 48us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.719028 (+ 61us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.719119 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260319 16:33:43.719218 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.713067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34748 (local address 127.24.187.253:35849)
0319 16:33:43.713121 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.713123 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.713158 (+ 35us) server_negotiation.cc:408] Connection header received
0319 16:33:43.713241 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.713242 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.713254 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.713305 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.713692 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.714479 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.715028 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.715192 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.716671 (+ 1479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.716675 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.716675 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.716683 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.718088 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.718643 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.718644 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.718645 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.718667 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.718787 (+ 120us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.718788 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.718789 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.718871 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.718924 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.719057 (+ 133us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.719137 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260319 16:33:43.719573 344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.252@KRBTEST.COM for oryx/127.24.187.252@KRBTEST.COM
I20260319 16:33:43.724785 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.719801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53706)
0319 16:33:43.719839 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.719848 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.719880 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.719977 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.719977 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.720083 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.720194 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.720196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.720808 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.720809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.721125 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.721126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.721187 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.721550 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.721554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.722679 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.724048 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.724049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.724049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.724175 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.724370 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.724371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.724371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.724402 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.724566 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.724567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.724608 (+ 41us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.724653 (+ 45us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.724728 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260319 16:33:43.724864 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.719851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.24.187.252:42251)
0319 16:33:43.719888 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.719890 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.719898 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:43.719916 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.719916 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.719925 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.719974 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.720281 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.720705 (+ 424us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.721198 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.721322 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.722749 (+ 1427us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.722751 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.722752 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.722757 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.723968 (+ 1211us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.724255 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.724256 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.724257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.724278 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.724457 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.724457 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.724457 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.724527 (+ 70us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.724569 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.724670 (+ 101us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.724778 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":21}
I20260319 16:33:43.726040 344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Bootstrap starting.
I20260319 16:33:43.726558 344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:43.726965 344 log.cc:826] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:43.727696 344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: No bootstrap required, opened a new log
I20260319 16:33:43.728885 344 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } }
I20260319 16:33:43.729008 344 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:43.729030 344 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc58ffbf41b424e8e84eb71d38bd07c, State: Initialized, Role: FOLLOWER
I20260319 16:33:43.729128 344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } }
I20260319 16:33:43.729405 344 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:43.729380 349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } } }
I20260319 16:33:43.729462 349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: This master's current role is: FOLLOWER
W20260319 16:33:43.731145 360 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:33:43.731200 360 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:33:43.733165 32744 master_runner.cc:428] Detected that this master 8fc58ffbf41b424e8e84eb71d38bd07c is joining an existing cluster
I20260319 16:33:43.733209 32744 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.187.252:42251
I20260319 16:33:43.755990 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.734380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.24.187.252:42251)
0319 16:33:43.734435 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.734437 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.735020 (+ 583us) server_negotiation.cc:408] Connection header received
0319 16:33:43.735064 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.735065 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.735072 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.735113 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.750443 (+ 15330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.751132 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.751851 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.752006 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.753229 (+ 1223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.753232 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.753233 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.753239 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.754904 (+ 1665us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755338 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.755339 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.755339 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.755360 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755491 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.755491 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.755492 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.755600 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.755639 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.755806 (+ 167us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.755889 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:33:43.756053 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.734150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.24.187.254:45081)
0319 16:33:43.734191 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.734193 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.734507 (+ 314us) server_negotiation.cc:408] Connection header received
0319 16:33:43.734565 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.734566 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.734578 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.734638 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.750474 (+ 15836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.751186 (+ 712us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.751988 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.752156 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.753398 (+ 1242us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.753402 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.753403 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.753410 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.754943 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755365 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.755366 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.755366 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.755391 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755542 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.755542 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.755542 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.755656 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.755696 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.755877 (+ 181us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.755958 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:33:43.756248 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.734276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53714)
0319 16:33:43.734945 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.734954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.735015 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.735119 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.735120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.750019 (+ 14899us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.750312 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.750315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.751225 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.751226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.751771 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.751773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.752051 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.752515 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.752520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.753131 (+ 611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.754980 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.754981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.754982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.755259 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.755415 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.755416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.755439 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.755676 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.755677 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.755712 (+ 35us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.756073 (+ 361us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.756156 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":64,"threads_started":1}
I20260319 16:33:43.756696 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.734380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.24.187.253:35849)
0319 16:33:43.734438 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.734440 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.734574 (+ 134us) server_negotiation.cc:408] Connection header received
0319 16:33:43.734692 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.734693 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.734704 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.734737 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.750661 (+ 15924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.751336 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.752000 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.752160 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.753544 (+ 1384us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.753548 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.753549 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.753557 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.755732 (+ 2175us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.756061 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.756063 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.756063 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.756088 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.756235 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.756236 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.756236 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.756333 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.756375 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.756532 (+ 157us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.756613 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:33:43.756708 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.734184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34756)
0319 16:33:43.734488 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.734498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.734611 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.734812 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.734812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.750025 (+ 15213us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.750551 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.750553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.751417 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.751417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.751919 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.751920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.752233 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.752809 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.752812 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.753458 (+ 646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.755810 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.755811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.755985 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.756151 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.756152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.756152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.756177 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.756411 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.756411 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.756442 (+ 31us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.756493 (+ 51us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.756579 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":81,"threads_started":1}
I20260319 16:33:43.757362 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.734079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39856)
0319 16:33:43.734440 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.734450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.734508 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.734642 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.734643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.750032 (+ 15389us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.750308 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.750315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.751277 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.751278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.751887 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.751890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.752206 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.752747 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.752751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.753322 (+ 571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.755035 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.755037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.755285 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.755455 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.755456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.755456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.755483 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.755738 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.755739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.755778 (+ 39us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.756730 (+ 952us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.757265 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":86,"threads_started":1}
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938022, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.254@KRBTEST.COM for oryx/127.24.187.252@KRBTEST.COM
I20260319 16:33:43.764820 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.759212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.24.187.252:42251)
0319 16:33:43.759254 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.759256 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.759264 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:43.759314 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.759315 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.759322 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.759407 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.759914 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.760329 (+ 415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.761050 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.761181 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.762799 (+ 1618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.762802 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.762802 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.762808 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.764008 (+ 1200us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.764313 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.764314 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.764314 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.764329 (+ 15us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.764447 (+ 118us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.764447 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.764448 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.764517 (+ 69us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.764560 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.764702 (+ 142us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.764761 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:33:43.764876 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.759151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53722)
0319 16:33:43.759201 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.759211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.759252 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.759455 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.759457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.759631 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.759831 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.759833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.760428 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.760429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.760966 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.760968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.761041 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.761390 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.761394 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.762725 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.764079 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.764080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.764080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.764252 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.764374 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.764374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.764375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.764402 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.764564 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.764565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.764606 (+ 41us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.764677 (+ 71us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.764778 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260319 16:33:43.765412 32081 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.24.187.252:42251: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c"
member_type: NON_VOTER
last_known_addr {
host: "127.24.187.252"
port: 42251
}
attrs {
promote: true
}
}
dest_uuid: "41f56935045b4f47be93f73ad5537b88"
cas_config_opid_index: 5
I20260319 16:33:43.765654 32081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } }
I20260319 16:33:43.766039 32650 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.766345 318 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:43.767043 32650 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.24.187.252:42251
I20260319 16:33:43.767114 32650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.767171 32650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:43.767196 318 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Refusing update from remote peer 41f56935045b4f47be93f73ad5537b88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260319 16:33:43.767270 32744 master.cc:561] Master@127.24.187.252:42251 shutting down...
I20260319 16:33:43.767580 32648 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260319 16:33:43.768242 32066 consensus_peers.cc:597] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 -> Peer 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252:42251): Couldn't send request to peer 8fc58ffbf41b424e8e84eb71d38bd07c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260319 16:33:43.770774 32744 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:43.770828 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.766166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34772)
0319 16:33:43.767926 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.767939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.767988 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.768201 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.768203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.768204 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:43.768381 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.768382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.769407 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.769408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.770534 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.770537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.770604 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.770682 (+ 78us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.770718 (+ 36us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.770763 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1742}
I20260319 16:33:43.770864 32744 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:43.770907 32744 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: stopping tablet replica
I20260319 16:33:43.771584 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.766217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.24.187.253:35849)
0319 16:33:43.766258 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.766260 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.768038 (+ 1778us) server_negotiation.cc:408] Connection header received
0319 16:33:43.768066 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.768067 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.768095 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.768141 (+ 46us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:43.768462 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.769312 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.770911 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.771372 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.771463 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.771485 (+ 22us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.771520 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:33:43.771782 32698 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 LEARNER]: Refusing update from remote peer 41f56935045b4f47be93f73ad5537b88: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260319 16:33:43.771991 32031 connection.cc:537] client connection to 127.24.187.252:42251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260319 16:33:43.772105 32648 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260319 16:33:43.772683 32698 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.773846 32648 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: attempting to promote NON_VOTER 919c26ab3f984ce6a3546c6ce1131317 to VOTER
I20260319 16:33:43.773857 370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.773934 370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: LEARNER
I20260319 16:33:43.773896 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f56935045b4f47be93f73ad5537b88. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: NON_VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: true } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.773978 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: LEARNER
I20260319 16:33:43.774051 32648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } }
I20260319 16:33:43.774391 32698 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 LEARNER]: Refusing update from remote peer 41f56935045b4f47be93f73ad5537b88: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260319 16:33:43.774556 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.774371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53730)
0319 16:33:43.774426 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.774526 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260319 16:33:43.774789 32650 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260319 16:33:43.775009 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.774906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53734)
0319 16:33:43.774946 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.774983 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260319 16:33:43.775064 32066 consensus_peers.cc:597] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 -> Peer 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252:42251): Couldn't send request to peer 8fc58ffbf41b424e8e84eb71d38bd07c. Status: Network error: Client connection negotiation failed: client connection to 127.24.187.252:42251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260319 16:33:43.775396 32649 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 919c26ab3f984ce6a3546c6ce1131317 (127.24.187.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.776173 32649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.776249 32649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:43.776186 32650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.776300 32650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:43.776464 32698 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 919c26ab3f984ce6a3546c6ce1131317 (127.24.187.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.777103 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.777184 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:33:43.783623 32744 master.cc:583] Master@127.24.187.252:42251 shutdown complete.
I20260319 16:33:43.783689 32744 master_runner.cc:305] Clearing existing system tablet
I20260319 16:33:43.783823 32744 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.783870 32744 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.784556 32744 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:43.785087 372 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.785259 32744 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.785316 32744 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.785339 32744 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
uuid: "8fc58ffbf41b424e8e84eb71d38bd07c"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "24461b1d74c8160ccd356fe492d9d426"
server_key_iv: "a68c545954e599ed6b5e8b3a68ab6eb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.786118 32744 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:43.787509 32744 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260319 16:33:43.787572 32744 log.cc:1199] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260319 16:33:43.787755 32744 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Deleting consensus metadata
I20260319 16:33:43.787915 32744 master_runner.cc:315] Copying system tablet from 127.24.187.254:45081
I20260319 16:33:43.788568 32744 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.187.254:45081
I20260319 16:33:43.794659 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.788930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39860)
0319 16:33:43.789074 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.789086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.789155 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.789303 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.789303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.789495 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:43.789878 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.789881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.790832 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.790833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.791321 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.791324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.791419 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.791854 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:43.791858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:43.792105 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:43.793807 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.793808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.793809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.793954 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.794102 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:43.794103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:43.794103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:43.794134 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:43.794382 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:43.794383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:43.794428 (+ 45us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.794492 (+ 64us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.794576 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":99,"threads_started":1}
I20260319 16:33:43.794710 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.789030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.24.187.254:45081)
0319 16:33:43.789082 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.789084 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.789153 (+ 69us) server_negotiation.cc:408] Connection header received
0319 16:33:43.789206 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.789207 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.789218 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.789293 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.789959 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.790739 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.791436 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.791604 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.792175 (+ 571us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.792179 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.792179 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.792186 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.793668 (+ 1482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.794016 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.794017 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.794018 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.794043 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.794190 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.794190 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.794190 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.794295 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.794361 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.794530 (+ 169us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.794608 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:43.795078 32101 tablet_copy_service.cc:140] P 41f56935045b4f47be93f73ad5537b88: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8fc58ffbf41b424e8e84eb71d38bd07c ({username='oryx', principal='oryx/127.24.187.252@KRBTEST.COM'} at 127.0.0.1:39860)
I20260319 16:33:43.795143 32101 tablet_copy_service.cc:161] P 41f56935045b4f47be93f73ad5537b88: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8fc58ffbf41b424e8e84eb71d38bd07c at {username='oryx', principal='oryx/127.24.187.252@KRBTEST.COM'} at 127.0.0.1:39860: session id = 8fc58ffbf41b424e8e84eb71d38bd07c-00000000000000000000000000000000
I20260319 16:33:43.795639 32101 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: Tablet Copy: opened 0 blocks and 1 log segments
I20260319 16:33:43.796087 32744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260319 16:33:43.797363 32744 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: tablet copy: Starting download of 0 data blocks...
I20260319 16:33:43.797498 32744 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: tablet copy: Starting download of 1 WAL segments...
I20260319 16:33:43.798399 32744 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260319 16:33:43.799213 32101 tablet_copy_service.cc:342] P 41f56935045b4f47be93f73ad5537b88: Request end of tablet copy session 8fc58ffbf41b424e8e84eb71d38bd07c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.187.252@KRBTEST.COM'} at 127.0.0.1:39860
I20260319 16:33:43.799271 32101 tablet_copy_service.cc:434] P 41f56935045b4f47be93f73ad5537b88: ending tablet copy session 8fc58ffbf41b424e8e84eb71d38bd07c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8fc58ffbf41b424e8e84eb71d38bd07c
W20260319 16:33:43.800854 32744 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260319 16:33:43.801620 32744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:43.802821 379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.802860 382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:43.803020 380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:43.803094 32744 server_base.cc:1060] running on GCE node
I20260319 16:33:43.803176 32744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:43.803303 32744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:43.804450 32744 hybrid_clock.cc:648] HybridClock initialized: now 1773938023804440 us; error 25 us; skew 500 ppm
Mar 19 16:33:43 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938023, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:43.805742 32744 init.cc:377] Logged in from keytab as oryx/127.24.187.252@KRBTEST.COM (short username oryx)
I20260319 16:33:43.805997 32744 webserver.cc:492] Webserver started at http://127.24.187.252:46039/ using document root <none> and password file <none>
I20260319 16:33:43.806097 32744 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:43.806129 32744 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:43.806797 32744 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:43.807233 389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.807394 32744 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:43.807437 32744 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
uuid: "8fc58ffbf41b424e8e84eb71d38bd07c"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "24461b1d74c8160ccd356fe492d9d426"
server_key_iv: "a68c545954e599ed6b5e8b3a68ab6eb2"
server_key_version: "encryptionkey@0"
I20260319 16:33:43.807490 32744 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:43.814558 32744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:43.815873 32744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:43.818858 32744 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.252:42251
I20260319 16:33:43.818892 441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.252:42251 every 8 connection(s)
I20260319 16:33:43.819172 32744 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
I20260319 16:33:43.819784 442 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:33:43.820150 442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Bootstrap starting.
I20260319 16:33:43.822319 442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:33:43.822510 442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: Bootstrap complete.
I20260319 16:33:43.822739 442 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } }
I20260319 16:33:43.822827 442 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fc58ffbf41b424e8e84eb71d38bd07c, State: Initialized, Role: LEARNER
I20260319 16:33:43.822870 442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } }
I20260319 16:33:43.823012 442 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:43.824445 457 catalog_manager.cc:1269] Loaded cluster ID: a509a400332f44e79b72a654d1fa34e2
I20260319 16:33:43.824290 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:43.824623 457 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: loading cluster ID for follower catalog manager: success
I20260319 16:33:43.824640 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: This master's current role is: LEARNER
I20260319 16:33:43.825925 457 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: acquiring CA information for follower catalog manager: success
I20260319 16:33:43.826320 457 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260319 16:33:43.829427 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.822718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.24.187.252:42251)
0319 16:33:43.822861 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.822863 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.822876 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:33:43.822899 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.822900 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.822911 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.823011 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:43.823844 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.824868 (+ 1024us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.825681 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.825865 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.826726 (+ 861us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:43.826729 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:43.826730 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:43.826736 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:43.828416 (+ 1680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.828725 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.828726 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.828727 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.828750 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:43.828865 (+ 115us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:43.828866 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:43.828866 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:43.828964 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:43.829048 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.829231 (+ 183us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.829308 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260319 16:33:43.832412 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.830456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39866)
0319 16:33:43.830494 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.830504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.830547 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.830654 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.830654 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.830656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:43.830817 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.830819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.831691 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.831692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.832247 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.832249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.832307 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.832310 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.832338 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.832361 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260319 16:33:43.832931 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.830509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.24.187.254:45081)
0319 16:33:43.830546 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.830548 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.830556 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:43.830583 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.830584 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.830592 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.830654 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:43.830878 (+ 224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.831584 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.832332 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.832786 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.832820 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.832844 (+ 24us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.832875 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:33:43.835718 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.833986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53742)
0319 16:33:43.834032 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:43.834042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:43.834080 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:43.834287 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:43.834289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:43.834290 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:43.834416 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.834417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.834997 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.834998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:43.835534 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:43.835536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.835602 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.835605 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:43.835633 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:33:43.835661 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:33:43.836115 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:43.834150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.24.187.252:42251)
0319 16:33:43.834192 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:43.834193 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:43.834199 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:33:43.834215 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:43.834216 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:43.834229 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:43.834276 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:43.834471 (+ 195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.834933 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:43.835610 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:43.835990 (+ 380us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:43.836037 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:43.836056 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:33:43.836078 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:33:43.901324 32274 heartbeater.cc:499] Master 127.24.187.254:45081 was elected leader, sending a full tablet report...
I20260319 16:33:44.055007 32410 heartbeater.cc:499] Master 127.24.187.254:45081 was elected leader, sending a full tablet report...
I20260319 16:33:44.181456 416 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260319 16:33:44.181735 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f56935045b4f47be93f73ad5537b88. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:44.181845 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: This master's current role is: LEARNER
I20260319 16:33:44.210904 32546 heartbeater.cc:499] Master 127.24.187.254:45081 was elected leader, sending a full tablet report...
I20260319 16:33:44.662226 32648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: NON_VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: true } } }
I20260319 16:33:44.662429 32648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:44.662767 32648 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: attempting to promote NON_VOTER 8fc58ffbf41b424e8e84eb71d38bd07c to VOTER
I20260319 16:33:44.662890 32648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } }
I20260319 16:33:44.663290 416 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 LEARNER]: Refusing update from remote peer 41f56935045b4f47be93f73ad5537b88: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260319 16:33:44.663338 32698 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Refusing update from remote peer 41f56935045b4f47be93f73ad5537b88: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260319 16:33:44.663501 460 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260319 16:33:44.663635 458 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260319 16:33:44.664359 458 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.664579 32698 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.664886 416 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.665558 32648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.665628 32648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:44.665482 462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.665627 460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.665755 462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:33:44.665772 460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:44.665881 458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.665961 458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:44.665987 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f56935045b4f47be93f73ad5537b88" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:44.666049 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:33:44.883289 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.881229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39880)
0319 16:33:44.881351 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:44.881364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:44.881433 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:44.881666 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:44.881668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:44.881670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:44.881869 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.881870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.882541 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.882542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:44.883117 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.883118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.883173 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.883176 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:44.883196 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:33:44.883228 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260319 16:33:44.883683 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.881283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.24.187.254:45081)
0319 16:33:44.881527 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:44.881529 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:44.881541 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:33:44.881561 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:44.881562 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:44.881571 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:44.881634 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:44.881945 (+ 311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.882484 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.883187 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.883580 (+ 393us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.883588 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:44.883606 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:33:44.883634 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":166,"threads_started":1}
I20260319 16:33:44.886142 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.884259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34782)
0319 16:33:44.884309 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:44.884320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:44.884350 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:44.884516 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:44.884516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:44.884518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:44.884634 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.884649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.885335 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.885336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:44.885968 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.885970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.886032 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.886035 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:44.886058 (+ 23us) client_negotiation.cc:240] Negotiation successful
0319 16:33:44.886091 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260319 16:33:44.886514 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.884310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.24.187.253:35849)
0319 16:33:44.884392 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:44.884394 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:44.884400 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:33:44.884416 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:44.884417 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:44.884449 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:44.884504 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:44.884727 (+ 223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.885242 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.886042 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.886414 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.886421 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:44.886442 (+ 21us) server_negotiation.cc:300] Negotiation successful
0319 16:33:44.886470 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":45,"threads_started":1}
I20260319 16:33:44.888810 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.887021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53756)
0319 16:33:44.887060 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:44.887070 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:44.887105 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:44.887378 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:44.887379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:44.887380 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:44.887488 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.887489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.888146 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.888147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:44.888666 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.888668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.888731 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.888734 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:44.888748 (+ 14us) client_negotiation.cc:240] Negotiation successful
0319 16:33:44.888771 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260319 16:33:44.889290 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.887131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.24.187.252:42251)
0319 16:33:44.887271 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:44.887273 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:44.887281 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:44.887301 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:44.887302 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:44.887311 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:44.887367 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:44.887537 (+ 170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.888075 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.888784 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.889187 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.889194 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:44.889213 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:33:44.889242 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260319 16:33:44.892275 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.890543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:39884)
0319 16:33:44.890659 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:44.890669 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:44.890731 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:44.890834 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:44.890834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:44.890835 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:44.890969 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.890970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.891553 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.891553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:44.892130 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.892132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.892185 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.892188 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:44.892204 (+ 16us) client_negotiation.cc:240] Negotiation successful
0319 16:33:44.892229 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260319 16:33:44.892719 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.890664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.24.187.254:45081)
0319 16:33:44.890721 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:44.890722 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:44.890732 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:44.890782 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:44.890783 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:44.890790 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:44.890832 (+ 42us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:44.891018 (+ 186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.891484 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.892228 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.892627 (+ 399us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.892634 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:44.892654 (+ 20us) server_negotiation.cc:300] Negotiation successful
0319 16:33:44.892675 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260319 16:33:44.894786 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.893237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.0.0.1:34788)
0319 16:33:44.893275 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:44.893286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:44.893334 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:44.893434 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:44.893435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:44.893437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:44.893547 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.893548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.894120 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.894121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:44.894647 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.894648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.894701 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.894703 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:44.894723 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:33:44.894744 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260319 16:33:44.895150 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.893305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.24.187.253:35849)
0319 16:33:44.893358 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:44.893360 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:44.893366 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:33:44.893383 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:44.893383 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:44.893390 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:44.893432 (+ 42us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:44.893597 (+ 165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.894070 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.894725 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.895070 (+ 345us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.895076 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:44.895091 (+ 15us) server_negotiation.cc:300] Negotiation successful
0319 16:33:44.895111 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260319 16:33:44.897185 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.895669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.0.0.1:53770)
0319 16:33:44.895737 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:44.895747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:44.895788 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:44.895876 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:44.895877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:44.895878 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:44.895983 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.895983 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.896515 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.896516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:44.897047 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:44.897048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.897101 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.897103 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:44.897123 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:33:44.897145 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260319 16:33:44.897606 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:44.895749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.24.187.252:42251)
0319 16:33:44.895791 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:44.895793 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:44.895812 (+ 19us) server_negotiation.cc:408] Connection header received
0319 16:33:44.895827 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:44.895827 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:44.895833 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:44.895866 (+ 33us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:44.896029 (+ 163us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.896457 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:44.897142 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:44.897513 (+ 371us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:44.897519 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:44.897538 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:33:44.897563 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:44.898510 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 32145
I20260319 16:33:44.903596 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:36825
--local_ip_for_outbound_sockets=127.24.187.193
--tserver_master_addrs=127.24.187.254:45081
--webserver_port=46651
--webserver_interface=127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:44.976903 469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:44.977074 469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:44.977097 469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:44.978585 469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:44.978636 469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:44.978660 469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:44.980263 469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:36825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=46651
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:44.980445 469 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:44.980613 469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:44.983002 474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:44.983065 469 server_base.cc:1060] running on GCE node
W20260319 16:33:44.983031 475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:44.983006 477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:44.983319 469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:44.983536 469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:44.984701 469 hybrid_clock.cc:648] HybridClock initialized: now 1773938024984681 us; error 34 us; skew 500 ppm
Mar 19 16:33:44 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938024, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:44.987320 469 init.cc:377] Logged in from keytab as oryx/127.24.187.193@KRBTEST.COM (short username oryx)
I20260319 16:33:44.987835 469 webserver.cc:492] Webserver started at http://127.24.187.193:46651/ using document root <none> and password file <none>
I20260319 16:33:44.988024 469 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:44.988085 469 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:44.989307 469 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:44.989948 484 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:44.990134 469 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:44.990196 469 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "595ada663e79499f80cb25638ccf6ebb"
format_stamp: "Formatted at 2026-03-19 16:33:42 on dist-test-slave-svn8"
server_key: "d2439b24bb75574942512c5bfd35a091"
server_key_iv: "3f22b179e4bf44941e12ad4631c3a2f0"
server_key_version: "encryptionkey@0"
I20260319 16:33:44.990487 469 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:45.002699 469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:45.004484 469 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:45.004608 469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:45.004834 469 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:45.005112 469 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:45.005142 469 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.005170 469 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:45.005194 469 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.010998 469 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:36825
I20260319 16:33:45.011054 597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:36825 every 8 connection(s)
I20260319 16:33:45.011338 469 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:45.018038 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 469
I20260319 16:33:45.018186 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 32281
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938024, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938024, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.253@KRBTEST.COM
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938024, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.252@KRBTEST.COM
I20260319 16:33:45.025333 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.012224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:55803 (local address 127.24.187.253:35849)
0319 16:33:45.012272 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.012274 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.012497 (+ 223us) server_negotiation.cc:408] Connection header received
0319 16:33:45.013093 (+ 596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.013095 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.013108 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.013140 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.014241 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.014739 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.015521 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.015693 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.022125 (+ 6432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.022129 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.022129 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.022137 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.023979 (+ 1842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024389 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.024391 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.024391 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.024414 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024604 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.024605 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.024605 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.024739 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.024794 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.025143 (+ 349us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.025243 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:33:45.025342 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.012616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:52449 (local address 127.24.187.252:42251)
0319 16:33:45.012675 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.012676 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.012688 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:33:45.013224 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.013226 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.013238 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.013288 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.014747 (+ 1459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.015474 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.016012 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.016132 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.022026 (+ 5894us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.022031 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.022032 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.022039 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.023842 (+ 1803us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024383 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.024384 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.024384 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.024419 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024622 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.024623 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.024623 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.024753 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.024794 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.025147 (+ 353us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.025248 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260319 16:33:45.025342 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.012462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:34903 (local address 127.24.187.254:45081)
0319 16:33:45.012502 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.012504 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.012513 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:45.013099 (+ 586us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.013100 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.013111 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.013161 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.014159 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.014670 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.015231 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.015400 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.022027 (+ 6627us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.022032 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.022032 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.022040 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.024303 (+ 2263us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024609 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.024610 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.024610 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.024634 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024794 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.024794 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.024795 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.024893 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.024930 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.025147 (+ 217us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.025249 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:33:45.025491 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:46733
--local_ip_for_outbound_sockets=127.24.187.194
--tserver_master_addrs=127.24.187.254:45081
--webserver_port=35605
--webserver_interface=127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:33:45.025841 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.012217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.24.187.193:55803)
0319 16:33:45.012380 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.012415 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.012982 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.013245 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.013246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.013638 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.014156 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.014158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.014816 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.014817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.015403 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.015404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.015503 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.018942 (+ 3439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.018947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.022016 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.024088 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.024089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.024316 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.024472 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.024473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.024549 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.024851 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.024856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.024996 (+ 140us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.025453 (+ 457us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.025606 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260319 16:33:45.025856 604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.012531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.24.187.193:52449)
0319 16:33:45.012643 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.012651 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.012971 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.013305 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.013306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.013607 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.014065 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.014068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.015554 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.015554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.015946 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.015948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.016001 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.019633 (+ 3632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.019637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.021910 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.023956 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.023962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.023965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.024296 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.024483 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.024484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.024553 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.024849 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.024854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.024996 (+ 142us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.025460 (+ 464us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.025681 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":74,"thread_start_us":58,"threads_started":1}
I20260319 16:33:45.026270 599 heartbeater.cc:344] Connected to a master server at 127.24.187.252:42251
I20260319 16:33:45.026351 599 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.026538 599 heartbeater.cc:507] Master 127.24.187.252:42251 requested a full tablet report, sending...
I20260319 16:33:45.027004 406 ts_manager.cc:194] Registered new tserver with Master: 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:45.027299 600 heartbeater.cc:344] Connected to a master server at 127.24.187.253:35849
I20260319 16:33:45.027364 600 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.027464 600 heartbeater.cc:507] Master 127.24.187.253:35849 requested a full tablet report, sending...
I20260319 16:33:45.027559 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.012216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.193:34903)
0319 16:33:45.012424 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.012434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.012952 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.013214 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.013217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.013625 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.014065 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.014068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.014756 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.014757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.015159 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.015160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.015266 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.018890 (+ 3624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.018914 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.021930 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.024382 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.024383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.024532 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.024708 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.024709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.024709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.024734 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.024970 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.024971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.025000 (+ 29us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.027339 (+ 2339us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.027444 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260319 16:33:45.027839 32688 ts_manager.cc:194] Registered new tserver with Master: 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:45.027935 598 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45081
I20260319 16:33:45.027989 598 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.028151 598 heartbeater.cc:507] Master 127.24.187.254:45081 requested a full tablet report, sending...
I20260319 16:33:45.028429 32081 ts_manager.cc:194] Re-registered known tserver with Master: 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:45.028784 32081 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:34903
W20260319 16:33:45.100703 605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:45.100883 605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:45.100905 605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:45.102409 605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:45.102458 605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:45.102483 605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:45.104019 605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:46733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=35605
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:45.104205 605 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:45.104336 605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:45.106680 613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:45.106772 610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:45.106864 605 server_base.cc:1060] running on GCE node
W20260319 16:33:45.106913 611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:45.107111 605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:45.107285 605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:45.108426 605 hybrid_clock.cc:648] HybridClock initialized: now 1773938025108405 us; error 37 us; skew 500 ppm
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:45.110795 605 init.cc:377] Logged in from keytab as oryx/127.24.187.194@KRBTEST.COM (short username oryx)
I20260319 16:33:45.111275 605 webserver.cc:492] Webserver started at http://127.24.187.194:35605/ using document root <none> and password file <none>
I20260319 16:33:45.111421 605 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:45.111459 605 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:45.112675 605 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:45.113204 620 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.113364 605 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:45.113427 605 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "58d9dd27bba64c15963825d9e53958dc"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "e923b5d6f6198a6070b1d90db311c1fe"
server_key_iv: "08ace35a5ec3497a3c446e5d9833d053"
server_key_version: "encryptionkey@0"
I20260319 16:33:45.113696 605 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:45.137195 605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:45.139142 605 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:45.139254 605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:45.139483 605 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:45.139786 605 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:45.139818 605 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.139840 605 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:45.139851 605 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.145141 605 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:46733
I20260319 16:33:45.145304 733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:46733 every 8 connection(s)
I20260319 16:33:45.145480 605 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:45.150722 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 605
I20260319 16:33:45.150858 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 32417
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.253@KRBTEST.COM
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.252@KRBTEST.COM
I20260319 16:33:45.155265 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.146244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:33817 (local address 127.24.187.253:35849)
0319 16:33:45.146303 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.146305 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.147112 (+ 807us) server_negotiation.cc:408] Connection header received
0319 16:33:45.147145 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.147147 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.147162 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.147214 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.148462 (+ 1248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149004 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.149560 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149743 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.152258 (+ 2515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.152262 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.152263 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.152270 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.153936 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.154338 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.154339 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.154340 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.154363 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.154586 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.154587 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.154588 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.154721 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.154804 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.155084 (+ 280us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.155173 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:33:45.155606 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.146136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.24.187.194:33817)
0319 16:33:45.147037 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.147045 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.147088 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.147230 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.147231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.147645 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.148367 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.148369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149078 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.149079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.149469 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.149471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149790 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.150192 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.150196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.152157 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.154020 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.154025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.154028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.154264 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.154423 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.154424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.154424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.154490 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.154795 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.154799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.154942 (+ 143us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.155053 (+ 111us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.155187 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260319 16:33:45.156092 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.146136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:56761)
0319 16:33:45.146352 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.146427 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.146897 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.147137 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.147141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.147636 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.147995 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.147999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.148838 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.148838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.149345 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.149347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149644 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.150083 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.150086 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.152158 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.154327 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.154328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.154328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.155305 (+ 977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.155486 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.155486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.155487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.155513 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.155761 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.155762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.155800 (+ 38us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.155886 (+ 86us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.155992 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260319 16:33:45.156128 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.146251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:56761 (local address 127.24.187.254:45081)
0319 16:33:45.146303 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.146305 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.146497 (+ 192us) server_negotiation.cc:408] Connection header received
0319 16:33:45.146984 (+ 487us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.146986 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.146997 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.147063 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.148124 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.148749 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.149438 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149594 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.152262 (+ 2668us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.152266 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.152266 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.152274 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.153947 (+ 1673us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.155383 (+ 1436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.155384 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.155384 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.155409 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.155575 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.155576 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.155576 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.155678 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.155719 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.155945 (+ 226us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.156041 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:45.156313 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.146443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:44083 (local address 127.24.187.252:42251)
0319 16:33:45.146480 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.146482 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.147331 (+ 849us) server_negotiation.cc:408] Connection header received
0319 16:33:45.147363 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.147364 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.147380 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.147424 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.148085 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.148661 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.149393 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149552 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.153085 (+ 3533us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.153090 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.153090 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.153098 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.155183 (+ 2085us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.155644 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.155645 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.155646 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.155669 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.155832 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.155833 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.155833 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.155932 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.155969 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.156148 (+ 179us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.156235 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:33:45.156512 734 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45081
I20260319 16:33:45.156615 734 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.156662 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.146401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.24.187.194:44083)
0319 16:33:45.147262 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.147271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.147319 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.147437 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.147437 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.147647 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.147993 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.147996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.148747 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.148748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.149308 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.149310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.149432 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.149979 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.149998 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.152714 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.155407 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.155408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.155408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.155570 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.155744 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.155745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.155746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.155771 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.156012 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.156013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.156047 (+ 34us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.156415 (+ 368us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.156509 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":63,"threads_started":1}
I20260319 16:33:45.156963 734 heartbeater.cc:507] Master 127.24.187.254:45081 requested a full tablet report, sending...
I20260319 16:33:45.157006 735 heartbeater.cc:344] Connected to a master server at 127.24.187.252:42251
I20260319 16:33:45.157068 735 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.157171 735 heartbeater.cc:507] Master 127.24.187.252:42251 requested a full tablet report, sending...
I20260319 16:33:45.157342 32081 ts_manager.cc:194] Re-registered known tserver with Master: 58d9dd27bba64c15963825d9e53958dc (127.24.187.194:46733)
I20260319 16:33:45.157495 406 ts_manager.cc:194] Registered new tserver with Master: 58d9dd27bba64c15963825d9e53958dc (127.24.187.194:46733)
I20260319 16:33:45.157966 32081 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:56761
I20260319 16:33:45.158463 736 heartbeater.cc:344] Connected to a master server at 127.24.187.253:35849
I20260319 16:33:45.158524 736 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.158566 736 heartbeater.cc:507] Master 127.24.187.253:35849 requested a full tablet report, sending...
I20260319 16:33:45.158685 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:45293
--local_ip_for_outbound_sockets=127.24.187.195
--tserver_master_addrs=127.24.187.254:45081
--webserver_port=39085
--webserver_interface=127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:33:45.158859 32688 ts_manager.cc:194] Registered new tserver with Master: 58d9dd27bba64c15963825d9e53958dc (127.24.187.194:46733)
W20260319 16:33:45.232560 741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:45.232730 741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:45.232753 741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:45.234298 741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:45.234344 741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:45.234369 741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:45.236006 741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:45293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=39085
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:45081,127.24.187.253:35849,127.24.187.252:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:45.236171 741 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:45.236338 741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:45.238672 749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:45.238722 746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:45.238843 747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:45.238945 741 server_base.cc:1060] running on GCE node
I20260319 16:33:45.239084 741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:45.239255 741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:45.240425 741 hybrid_clock.cc:648] HybridClock initialized: now 1773938025240406 us; error 26 us; skew 500 ppm
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:45.242836 741 init.cc:377] Logged in from keytab as oryx/127.24.187.195@KRBTEST.COM (short username oryx)
I20260319 16:33:45.243275 741 webserver.cc:492] Webserver started at http://127.24.187.195:39085/ using document root <none> and password file <none>
I20260319 16:33:45.243445 741 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:45.243486 741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:45.244617 741 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:45.245169 756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.245383 741 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:45.245452 741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633"
format_stamp: "Formatted at 2026-03-19 16:33:43 on dist-test-slave-svn8"
server_key: "800bf9287be4a3df5a6b59561a556b0a"
server_key_iv: "0dc91fa5e83d0df36cae9b41b83b2d60"
server_key_version: "encryptionkey@0"
I20260319 16:33:45.245759 741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:45.267647 741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:45.269470 741 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:45.269590 741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:45.269831 741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:45.270124 741 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:45.270155 741 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.270184 741 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:45.270200 741 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:45.275548 741 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:45293
I20260319 16:33:45.275624 869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:45293 every 8 connection(s)
I20260319 16:33:45.276007 741 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.252@KRBTEST.COM
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
Mar 19 16:33:45 dist-test-slave-svn8 krb5kdc[32033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938025, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.253@KRBTEST.COM
I20260319 16:33:45.282997 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 741
I20260319 16:33:45.285429 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.277285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:40961 (local address 127.24.187.252:42251)
0319 16:33:45.277329 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.277331 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.277338 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:33:45.277593 (+ 255us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.277595 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.277621 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.277649 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.278606 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279072 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.279673 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279843 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.282253 (+ 2410us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.282256 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.282257 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.282264 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.284049 (+ 1785us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.284516 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.284518 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.284518 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.284547 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.284785 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.284786 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.284787 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.284895 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.284937 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.285247 (+ 310us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.285336 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:33:45.285851 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.277052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:33621 (local address 127.24.187.254:45081)
0319 16:33:45.277101 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.277103 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.277113 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:45.277595 (+ 482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.277597 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.277609 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.277640 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.278594 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279358 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.280019 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.280195 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.283078 (+ 2883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.283082 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.283083 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.283093 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.284818 (+ 1725us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.285173 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.285174 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.285175 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.285197 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.285350 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.285351 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.285351 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.285448 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.285501 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.285672 (+ 171us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.285787 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260319 16:33:45.285856 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.276690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:35377 (local address 127.24.187.253:35849)
0319 16:33:45.276770 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.276772 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.276897 (+ 125us) server_negotiation.cc:408] Connection header received
0319 16:33:45.277522 (+ 625us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.277524 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.277536 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.277581 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.278684 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279390 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.280416 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.280589 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.283133 (+ 2544us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.283137 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.283138 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.283145 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.284750 (+ 1605us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.285134 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.285136 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.285136 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.285162 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.285297 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.285298 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.285298 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.285418 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.285479 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.285664 (+ 185us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.285771 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260319 16:33:45.285938 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.277102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:42251 (local address 127.24.187.195:40961)
0319 16:33:45.277211 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.277220 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.277480 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.277738 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.277739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.278070 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.278487 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.278492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279147 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.279148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.279591 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.279592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279709 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.280175 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.280190 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.282169 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.284142 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.284148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.284151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.284420 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.284629 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.284629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.284630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.284714 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.284982 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.284986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.285091 (+ 105us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.285558 (+ 467us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.285733 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260319 16:33:45.286123 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.276934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.195:33621)
0319 16:33:45.277050 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.277058 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.277480 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.277738 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.277739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.278077 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.278487 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.278492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279444 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.279445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.279943 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.279946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.280009 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.280326 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.280330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.282960 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.284890 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.284891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.284891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.285096 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.285259 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.285260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.285260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.285284 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.285528 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.285529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.285571 (+ 42us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.285913 (+ 342us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.286022 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260319 16:33:45.286410 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.276669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:35849 (local address 127.24.187.195:35377)
0319 16:33:45.276792 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.276818 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.277435 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.277650 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.277655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.278064 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:45.278573 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.278576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.279470 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.279471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.280341 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.280343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.280401 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.280815 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:45.280819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:45.282981 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:45.284839 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.284840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.284841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.285063 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.285226 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:45.285226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:45.285227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:45.285253 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:45.285509 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:45.285510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:45.285552 (+ 42us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.285924 (+ 372us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.286321 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":293,"thread_start_us":73,"threads_started":1}
I20260319 16:33:45.286571 871 heartbeater.cc:344] Connected to a master server at 127.24.187.252:42251
I20260319 16:33:45.286653 871 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.286707 870 heartbeater.cc:344] Connected to a master server at 127.24.187.254:45081
I20260319 16:33:45.286746 872 heartbeater.cc:344] Connected to a master server at 127.24.187.253:35849
I20260319 16:33:45.286749 870 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.286815 872 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:45.286844 871 heartbeater.cc:507] Master 127.24.187.252:42251 requested a full tablet report, sending...
I20260319 16:33:45.286895 872 heartbeater.cc:507] Master 127.24.187.253:35849 requested a full tablet report, sending...
I20260319 16:33:45.286895 870 heartbeater.cc:507] Master 127.24.187.254:45081 requested a full tablet report, sending...
I20260319 16:33:45.287252 32688 ts_manager.cc:194] Registered new tserver with Master: 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195:45293)
I20260319 16:33:45.287317 406 ts_manager.cc:194] Registered new tserver with Master: 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195:45293)
I20260319 16:33:45.287467 32081 ts_manager.cc:194] Re-registered known tserver with Master: 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195:45293)
I20260319 16:33:45.287899 32081 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:33621
I20260319 16:33:45.295312 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.286418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.24.187.253:35849)
0319 16:33:45.287122 (+ 704us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.287123 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.287618 (+ 495us) server_negotiation.cc:408] Connection header received
0319 16:33:45.287682 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.287683 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.287694 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.287746 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.288559 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.289203 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.290242 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.290399 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.291911 (+ 1512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.291913 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.291914 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.291920 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.294108 (+ 2188us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.294477 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.294479 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.294479 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.294506 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.294677 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.294678 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.294678 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.294786 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.294875 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.295129 (+ 254us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.295224 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684}
I20260319 16:33:45.295639 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.286237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.24.187.254:45081)
0319 16:33:45.286278 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.286279 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.287775 (+ 1496us) server_negotiation.cc:408] Connection header received
0319 16:33:45.287842 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.287842 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.287853 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.287898 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.289170 (+ 1272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.289752 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.290338 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.290482 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.291957 (+ 1475us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.291961 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.291962 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.291970 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.294523 (+ 2553us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.294838 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.294840 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.294840 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.294868 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.295019 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.295020 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.295021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.295149 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.295238 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.295462 (+ 224us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.295547 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260319 16:33:45.297245 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.287246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.24.187.252:42251)
0319 16:33:45.287276 (+ 30us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.287278 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.287915 (+ 637us) server_negotiation.cc:408] Connection header received
0319 16:33:45.287999 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.288000 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.288008 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.288035 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.288880 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.289541 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.290257 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.290414 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.291872 (+ 1458us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.291875 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.291876 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.291882 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.296018 (+ 4136us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.296626 (+ 608us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.296628 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.296628 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.296667 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.296791 (+ 124us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.296792 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.296792 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.296904 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.296993 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.297083 (+ 90us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.297161 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260319 16:33:45.297649 32081 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:45.299458 32081 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:45.306427 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.303368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:45293 (local address 127.0.0.1:59278)
0319 16:33:45.303503 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.303514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.303568 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.304176 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.304176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.304178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:45.304407 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.304408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.305282 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.305283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.306199 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.306202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.306275 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.306280 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.306307 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.306351 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260319 16:33:45.306581 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.303550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:36825 (local address 127.0.0.1:58962)
0319 16:33:45.303835 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.303845 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.303895 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.304257 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.304258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.304259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:45.304451 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.304452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.305598 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.305599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.306313 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.306315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.306427 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.306431 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.306461 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.306500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":70,"threads_started":1}
I20260319 16:33:45.307247 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.303808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.24.187.195:45293)
0319 16:33:45.304019 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.304026 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.304036 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:45.304070 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.304072 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.304089 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.304480 (+ 391us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:45.304563 (+ 83us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.305203 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.306542 (+ 1339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.307044 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.307099 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.307131 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.307175 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260319 16:33:45.307271 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.303540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.24.187.194:46733)
0319 16:33:45.303739 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.303746 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.303757 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:45.303794 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.303797 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.303814 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.303870 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:45.304268 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.305331 (+ 1063us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.306628 (+ 1297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.307106 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.307140 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.307167 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.307198 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":122,"threads_started":1}
I20260319 16:33:45.306596 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.303455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46733 (local address 127.0.0.1:33456)
0319 16:33:45.303668 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.303678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.303720 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.303969 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.303970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.303972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:45.304181 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.304183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.305546 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.305547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.306400 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.306403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.306455 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.306458 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.306485 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.306522 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260319 16:33:45.307355 897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.303954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.24.187.193:36825)
0319 16:33:45.304123 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.304129 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.304138 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:45.304173 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.304175 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.304189 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.304247 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:45.304518 (+ 271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.305527 (+ 1009us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.306438 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.307196 (+ 758us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.307230 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.307256 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.307277 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260319 16:33:45.307792 804 tablet_service.cc:1511] Processing CreateTablet for tablet b6a14b999e944b978d8843aa141e6cc8 (DEFAULT_TABLE table=test-table [id=5d3415c48a354971b892e7453db176fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:45.307868 668 tablet_service.cc:1511] Processing CreateTablet for tablet b6a14b999e944b978d8843aa141e6cc8 (DEFAULT_TABLE table=test-table [id=5d3415c48a354971b892e7453db176fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:45.307868 532 tablet_service.cc:1511] Processing CreateTablet for tablet b6a14b999e944b978d8843aa141e6cc8 (DEFAULT_TABLE table=test-table [id=5d3415c48a354971b892e7453db176fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:45.308068 804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6a14b999e944b978d8843aa141e6cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:45.308104 532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6a14b999e944b978d8843aa141e6cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:45.308104 668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6a14b999e944b978d8843aa141e6cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:45.310946 898 tablet_bootstrap.cc:492] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Bootstrap starting.
I20260319 16:33:45.311189 899 tablet_bootstrap.cc:492] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Bootstrap starting.
I20260319 16:33:45.311805 898 tablet_bootstrap.cc:654] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:45.311954 899 tablet_bootstrap.cc:654] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:45.312095 898 log.cc:826] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:45.312191 899 log.cc:826] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:45.312618 900 tablet_bootstrap.cc:492] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Bootstrap starting.
I20260319 16:33:45.312836 898 tablet_bootstrap.cc:492] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: No bootstrap required, opened a new log
I20260319 16:33:45.312891 898 ts_tablet_manager.cc:1403] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:45.313346 900 tablet_bootstrap.cc:654] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:45.313560 900 log.cc:826] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:45.314018 899 tablet_bootstrap.cc:492] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: No bootstrap required, opened a new log
I20260319 16:33:45.314070 899 ts_tablet_manager.cc:1403] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260319 16:33:45.314085 898 raft_consensus.cc:359] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.314203 898 raft_consensus.cc:385] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:45.314239 898 raft_consensus.cc:740] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad8a9f1cb7e420dabb38fe28ae2d633, State: Initialized, Role: FOLLOWER
I20260319 16:33:45.314333 898 consensus_queue.cc:260] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.314540 898 ts_tablet_manager.cc:1434] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:45.314582 870 heartbeater.cc:499] Master 127.24.187.254:45081 was elected leader, sending a full tablet report...
I20260319 16:33:45.315428 899 raft_consensus.cc:359] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.315553 899 raft_consensus.cc:385] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:45.315603 899 raft_consensus.cc:740] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d9dd27bba64c15963825d9e53958dc, State: Initialized, Role: FOLLOWER
I20260319 16:33:45.315696 899 consensus_queue.cc:260] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.315909 899 ts_tablet_manager.cc:1434] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:45.315948 734 heartbeater.cc:499] Master 127.24.187.254:45081 was elected leader, sending a full tablet report...
I20260319 16:33:45.317437 900 tablet_bootstrap.cc:492] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: No bootstrap required, opened a new log
I20260319 16:33:45.317500 900 ts_tablet_manager.cc:1403] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260319 16:33:45.318662 900 raft_consensus.cc:359] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.318775 900 raft_consensus.cc:385] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:45.318796 900 raft_consensus.cc:740] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595ada663e79499f80cb25638ccf6ebb, State: Initialized, Role: FOLLOWER
I20260319 16:33:45.318866 900 consensus_queue.cc:260] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.319067 900 ts_tablet_manager.cc:1434] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260319 16:33:45.319132 598 heartbeater.cc:499] Master 127.24.187.254:45081 was elected leader, sending a full tablet report...
I20260319 16:33:45.337589 904 raft_consensus.cc:493] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:45.337733 904 raft_consensus.cc:515] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.338058 904 leader_election.cc:290] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58d9dd27bba64c15963825d9e53958dc (127.24.187.194:46733), 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:45.340118 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.338212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:36825 (local address 127.24.187.195:40909)
0319 16:33:45.338330 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.338339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.338379 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.338498 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.338499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.338501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:45.338697 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.338698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.339309 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.339310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.339896 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.339898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.339962 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.339967 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.339994 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.340035 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":72}
I20260319 16:33:45.340304 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.338212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46733 (local address 127.24.187.195:56643)
0319 16:33:45.338270 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:45.338282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:45.338336 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:45.338539 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:45.338540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:45.338541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:45.338697 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.338698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.339302 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.339303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:45.339898 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:45.339901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.339971 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.339975 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:45.340003 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:33:45.340039 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260319 16:33:45.340540 897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.338313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:40909 (local address 127.24.187.193:36825)
0319 16:33:45.338377 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.338379 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.338389 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:45.338425 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.338427 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.338436 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.338477 (+ 41us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:45.338768 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.339231 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.339976 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.340440 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.340449 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.340469 (+ 20us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.340495 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:33:45.340678 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.338306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:56643 (local address 127.24.187.194:46733)
0319 16:33:45.338424 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.338426 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.338434 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:45.338460 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.338461 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.338470 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.338534 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:45.338767 (+ 233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.339228 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.340091 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.340545 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.340554 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.340581 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.340616 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260319 16:33:45.340822 552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6a14b999e944b978d8843aa141e6cc8" candidate_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595ada663e79499f80cb25638ccf6ebb" is_pre_election: true
I20260319 16:33:45.340975 552 raft_consensus.cc:2468] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad8a9f1cb7e420dabb38fe28ae2d633 in term 0.
I20260319 16:33:45.340996 688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6a14b999e944b978d8843aa141e6cc8" candidate_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d9dd27bba64c15963825d9e53958dc" is_pre_election: true
I20260319 16:33:45.341123 688 raft_consensus.cc:2468] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad8a9f1cb7e420dabb38fe28ae2d633 in term 0.
I20260319 16:33:45.341200 757 leader_election.cc:304] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad8a9f1cb7e420dabb38fe28ae2d633, 595ada663e79499f80cb25638ccf6ebb; no voters:
I20260319 16:33:45.341316 904 raft_consensus.cc:2804] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:45.341369 904 raft_consensus.cc:493] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:45.341385 904 raft_consensus.cc:3060] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:45.342146 904 raft_consensus.cc:515] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.342268 904 leader_election.cc:290] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [CANDIDATE]: Term 1 election: Requested vote from peers 58d9dd27bba64c15963825d9e53958dc (127.24.187.194:46733), 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:45.342453 688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6a14b999e944b978d8843aa141e6cc8" candidate_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d9dd27bba64c15963825d9e53958dc"
I20260319 16:33:45.342497 552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6a14b999e944b978d8843aa141e6cc8" candidate_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595ada663e79499f80cb25638ccf6ebb"
I20260319 16:33:45.342558 552 raft_consensus.cc:3060] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:45.342558 688 raft_consensus.cc:3060] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:45.343400 552 raft_consensus.cc:2468] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad8a9f1cb7e420dabb38fe28ae2d633 in term 1.
I20260319 16:33:45.343400 688 raft_consensus.cc:2468] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad8a9f1cb7e420dabb38fe28ae2d633 in term 1.
I20260319 16:33:45.343597 758 leader_election.cc:304] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad8a9f1cb7e420dabb38fe28ae2d633, 58d9dd27bba64c15963825d9e53958dc; no voters:
I20260319 16:33:45.343691 904 raft_consensus.cc:2804] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:45.343829 904 raft_consensus.cc:697] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 LEADER]: Becoming Leader. State: Replica: 3ad8a9f1cb7e420dabb38fe28ae2d633, State: Running, Role: LEADER
I20260319 16:33:45.343911 904 consensus_queue.cc:237] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } }
I20260319 16:33:45.344767 32081 catalog_manager.cc:5671] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195). New cstate: current_term: 1 leader_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260319 16:33:45.353879 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.351999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.24.187.195:45293)
0319 16:33:45.352049 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.352050 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.352057 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:33:45.352088 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.352089 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.352103 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.352151 (+ 48us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:45.352372 (+ 221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.352841 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.353385 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.353549 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.353583 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:45.353666 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:45.353763 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.353787 (+ 24us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.353812 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260319 16:33:45.355370 688 raft_consensus.cc:1275] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 1 FOLLOWER]: Refusing update from remote peer 3ad8a9f1cb7e420dabb38fe28ae2d633: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:45.355369 552 raft_consensus.cc:1275] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Refusing update from remote peer 3ad8a9f1cb7e420dabb38fe28ae2d633: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:45.355619 904 consensus_queue.cc:1048] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:45.355734 907 consensus_queue.cc:1048] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:45.357959 910 mvcc.cc:204] Tried to move back new op lower bound from 7266050151853383680 to 7266050151809380352. Current Snapshot: MvccSnapshot[applied={T|T < 7266050151853383680}]
I20260319 16:33:45.358079 912 mvcc.cc:204] Tried to move back new op lower bound from 7266050151853383680 to 7266050151809380352. Current Snapshot: MvccSnapshot[applied={T|T < 7266050151853383680}]
I20260319 16:33:45.358585 911 mvcc.cc:204] Tried to move back new op lower bound from 7266050151853383680 to 7266050151809380352. Current Snapshot: MvccSnapshot[applied={T|T < 7266050151853383680}]
I20260319 16:33:45.360882 32080 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:45.360981 32080 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:45.362246 32080 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 41f56935045b4f47be93f73ad5537b88: Sending DeleteTablet for 3 replicas of tablet b6a14b999e944b978d8843aa141e6cc8
I20260319 16:33:45.362555 804 tablet_service.cc:1558] Processing DeleteTablet for tablet b6a14b999e944b978d8843aa141e6cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:45 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:59278
I20260319 16:33:45.362574 532 tablet_service.cc:1558] Processing DeleteTablet for tablet b6a14b999e944b978d8843aa141e6cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:45 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:58962
I20260319 16:33:45.362675 668 tablet_service.cc:1558] Processing DeleteTablet for tablet b6a14b999e944b978d8843aa141e6cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:45 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33456
I20260319 16:33:45.362922 920 tablet_replica.cc:333] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: stopping tablet replica
I20260319 16:33:45.363231 920 raft_consensus.cc:2243] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:45.363359 920 raft_consensus.cc:2272] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:45.363564 921 tablet_replica.cc:333] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: stopping tablet replica
I20260319 16:33:45.363596 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 32049
I20260319 16:33:45.364461 920 ts_tablet_manager.cc:1916] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:45.365667 919 tablet_replica.cc:333] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: stopping tablet replica
I20260319 16:33:45.365764 919 raft_consensus.cc:2243] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:45.365862 919 raft_consensus.cc:2272] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:45.365991 920 ts_tablet_manager.cc:1929] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:45.366056 920 log.cc:1199] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/b6a14b999e944b978d8843aa141e6cc8
I20260319 16:33:45.366264 920 ts_tablet_manager.cc:1950] T b6a14b999e944b978d8843aa141e6cc8 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Deleting consensus metadata
I20260319 16:33:45.366822 919 ts_tablet_manager.cc:1916] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:45.367929 919 ts_tablet_manager.cc:1929] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:45.367985 919 log.cc:1199] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/b6a14b999e944b978d8843aa141e6cc8
I20260319 16:33:45.368194 919 ts_tablet_manager.cc:1950] T b6a14b999e944b978d8843aa141e6cc8 P 595ada663e79499f80cb25638ccf6ebb: Deleting consensus metadata
I20260319 16:33:45.363646 921 raft_consensus.cc:2243] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:45.369863 921 raft_consensus.cc:2272] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:45.370224 921 ts_tablet_manager.cc:1916] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260319 16:33:45.370316 757 connection.cc:537] server connection from 127.0.0.1:59278 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260319 16:33:45.370407 486 connection.cc:537] server connection from 127.0.0.1:58962 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260319 16:33:45.371995 921 ts_tablet_manager.cc:1929] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:45.372056 921 log.cc:1199] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/b6a14b999e944b978d8843aa141e6cc8
I20260319 16:33:45.372267 921 ts_tablet_manager.cc:1950] T b6a14b999e944b978d8843aa141e6cc8 P 58d9dd27bba64c15963825d9e53958dc: Deleting consensus metadata
W20260319 16:33:45.372836 621 connection.cc:441] server connection from 127.0.0.1:33456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33456 (request call id 1) could send its response
I20260319 16:33:45.378887 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.372819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.24.187.252:42251)
0319 16:33:45.372856 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.372857 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.372966 (+ 109us) server_negotiation.cc:408] Connection header received
0319 16:33:45.373092 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.373093 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.373103 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.373148 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.373822 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.374285 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.375059 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.375199 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.376407 (+ 1208us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.376410 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.376411 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.376417 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.377855 (+ 1438us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.378209 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.378210 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.378210 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.378236 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.378395 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.378396 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.378396 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.378509 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.378614 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.378718 (+ 104us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.378796 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:33:45.379982 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:45.372024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.24.187.253:35849)
0319 16:33:45.372070 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:45.372072 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:45.372874 (+ 802us) server_negotiation.cc:408] Connection header received
0319 16:33:45.372981 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:45.372982 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:45.372994 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:45.373023 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:45.374336 (+ 1313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.375051 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:45.376148 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:45.376316 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:45.377367 (+ 1051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:45.377370 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:45.377371 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:45.377378 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:45.379000 (+ 1622us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.379485 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.379486 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.379486 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.379506 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:45.379619 (+ 113us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:45.379620 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:45.379620 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:45.379698 (+ 78us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:45.379761 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:45.379867 (+ 106us) server_negotiation.cc:300] Negotiation successful
0319 16:33:45.379924 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:33:46.359194 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.358805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.195:52599)
0319 16:33:46.358941 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.359124 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260319 16:33:46.359418 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.359204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:49907)
0319 16:33:46.359296 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.359370 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
W20260319 16:33:46.359442 870 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:45081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
I20260319 16:33:46.359678 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.359398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.193:34325)
0319 16:33:46.359564 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.359623 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":53,"threads_started":1}
W20260319 16:33:46.359700 734 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:45081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
W20260319 16:33:46.359951 598 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:45081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
I20260319 16:33:46.863130 932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41f56935045b4f47be93f73ad5537b88)
I20260319 16:33:46.863274 932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } }
I20260319 16:33:46.863571 932 leader_election.cc:290] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41f56935045b4f47be93f73ad5537b88 (127.24.187.254:45081), 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252:42251)
I20260319 16:33:46.863925 934 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41f56935045b4f47be93f73ad5537b88)
I20260319 16:33:46.863956 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.863750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52742)
0319 16:33:46.863855 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.863908 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
W20260319 16:33:46.864074 32673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
I20260319 16:33:46.864063 416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "919c26ab3f984ce6a3546c6ce1131317" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" is_pre_election: true
I20260319 16:33:46.864239 934 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } }
I20260319 16:33:46.864384 416 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 919c26ab3f984ce6a3546c6ce1131317 in term 1.
I20260319 16:33:46.864502 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.864399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52746)
0319 16:33:46.864437 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.864475 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260319 16:33:46.864579 32673 leader_election.cc:304] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fc58ffbf41b424e8e84eb71d38bd07c, 919c26ab3f984ce6a3546c6ce1131317; no voters:
I20260319 16:33:46.864681 934 leader_election.cc:290] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41f56935045b4f47be93f73ad5537b88 (127.24.187.254:45081), 919c26ab3f984ce6a3546c6ce1131317 (127.24.187.253:35849)
W20260319 16:33:46.864696 32673 leader_election.cc:336] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 41f56935045b4f47be93f73ad5537b88 (127.24.187.254:45081): Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
I20260319 16:33:46.864742 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.864549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52758)
0319 16:33:46.864651 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.864691 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":68,"threads_started":1}
I20260319 16:33:46.864768 932 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260319 16:33:46.864804 932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 41f56935045b4f47be93f73ad5537b88)
I20260319 16:33:46.864827 932 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 1 FOLLOWER]: Advancing to term 2
W20260319 16:33:46.864920 391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
I20260319 16:33:46.864907 32698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "919c26ab3f984ce6a3546c6ce1131317" is_pre_election: true
I20260319 16:33:46.865273 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.865190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52774)
0319 16:33:46.865231 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.865255 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260319 16:33:46.865352 391 leader_election.cc:336] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 41f56935045b4f47be93f73ad5537b88 (127.24.187.254:45081): Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
I20260319 16:33:46.865883 932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } }
I20260319 16:33:46.866005 932 leader_election.cc:290] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [CANDIDATE]: Term 2 election: Requested vote from peers 41f56935045b4f47be93f73ad5537b88 (127.24.187.254:45081), 8fc58ffbf41b424e8e84eb71d38bd07c (127.24.187.252:42251)
I20260319 16:33:46.866036 32698 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8fc58ffbf41b424e8e84eb71d38bd07c in current term 2: Already voted for candidate 919c26ab3f984ce6a3546c6ce1131317 in this term.
I20260319 16:33:46.866194 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.866091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52778)
0319 16:33:46.866139 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.866173 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260319 16:33:46.866231 416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "919c26ab3f984ce6a3546c6ce1131317" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c"
I20260319 16:33:46.866303 416 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 1 FOLLOWER]: Advancing to term 2
I20260319 16:33:46.866503 392 leader_election.cc:304] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fc58ffbf41b424e8e84eb71d38bd07c; no voters: 41f56935045b4f47be93f73ad5537b88, 919c26ab3f984ce6a3546c6ce1131317
I20260319 16:33:46.866565 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.866441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52780)
0319 16:33:46.866506 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.866543 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260319 16:33:46.866686 32673 leader_election.cc:336] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 41f56935045b4f47be93f73ad5537b88 (127.24.187.254:45081): Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
I20260319 16:33:46.867022 416 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 919c26ab3f984ce6a3546c6ce1131317 in term 2.
I20260319 16:33:46.867105 934 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260319 16:33:46.867177 32673 leader_election.cc:304] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8fc58ffbf41b424e8e84eb71d38bd07c, 919c26ab3f984ce6a3546c6ce1131317; no voters: 41f56935045b4f47be93f73ad5537b88
I20260319 16:33:46.867257 932 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 2 FOLLOWER]: Leader election won for term 2
I20260319 16:33:46.867347 932 raft_consensus.cc:697] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [term 2 LEADER]: Becoming Leader. State: Replica: 919c26ab3f984ce6a3546c6ce1131317, State: Running, Role: LEADER
I20260319 16:33:46.867403 932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } }
I20260319 16:33:46.867723 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919c26ab3f984ce6a3546c6ce1131317. Latest consensus state: current_term: 2 leader_uuid: "919c26ab3f984ce6a3546c6ce1131317" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:46.867807 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:46.867972 940 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:46.868292 940 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:46.868351 940 catalog_manager.cc:1269] Loaded cluster ID: a509a400332f44e79b72a654d1fa34e2
I20260319 16:33:46.868377 940 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:46.868537 940 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:46.868609 940 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Loaded TSK: 0
I20260319 16:33:46.868733 940 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:33:46.905875 32687 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:46.907078 32687 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:46.907990 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.907881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52798)
0319 16:33:46.907929 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.907959 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260319 16:33:46.908182 416 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [term 2 FOLLOWER]: Refusing update from remote peer 919c26ab3f984ce6a3546c6ce1131317: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260319 16:33:46.908313 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.908229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.0.0.1:52802)
0319 16:33:46.908260 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.908286 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260319 16:33:46.908414 32673 consensus_peers.cc:597] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 -> Peer 41f56935045b4f47be93f73ad5537b88 (127.24.187.254:45081): Couldn't send request to peer 41f56935045b4f47be93f73ad5537b88. Status: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260319 16:33:46.908484 932 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260319 16:33:46.909067 934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919c26ab3f984ce6a3546c6ce1131317. Latest consensus state: current_term: 2 leader_uuid: "919c26ab3f984ce6a3546c6ce1131317" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:46.909142 934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:33:46.909524 941 mvcc.cc:204] Tried to move back new op lower bound from 7266050158213312512 to 7266050158049439744. Current Snapshot: MvccSnapshot[applied={T|T < 7266050158213312512}]
I20260319 16:33:46.909653 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "919c26ab3f984ce6a3546c6ce1131317" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:46.909747 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:46.909701 942 mvcc.cc:204] Tried to move back new op lower bound from 7266050158213312512 to 7266050158049439744. Current Snapshot: MvccSnapshot[applied={T|T < 7266050158213312512}]
I20260319 16:33:46.909824 934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "919c26ab3f984ce6a3546c6ce1131317" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41f56935045b4f47be93f73ad5537b88" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 45081 } } peers { permanent_uuid: "919c26ab3f984ce6a3546c6ce1131317" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 35849 } attrs { promote: false } } peers { permanent_uuid: "8fc58ffbf41b424e8e84eb71d38bd07c" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 42251 } attrs { promote: false } } }
I20260319 16:33:46.909889 934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fc58ffbf41b424e8e84eb71d38bd07c [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:33:46.914222 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.911356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.24.187.194:46733)
0319 16:33:46.911722 (+ 366us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:46.911723 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:46.911731 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:46.911755 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:46.911756 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:46.911767 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:46.912159 (+ 392us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:46.912201 (+ 42us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.912911 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:46.913682 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.914123 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:46.914131 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:46.914149 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:33:46.914175 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":59,"threads_started":1}
I20260319 16:33:46.914429 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.911516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.24.187.193:36825)
0319 16:33:46.911631 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:46.911633 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:46.911639 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:33:46.911659 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:46.911659 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:46.911673 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:46.911725 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:46.912018 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.912527 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:46.913857 (+ 1330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.914303 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:46.914311 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:46.914334 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:33:46.914367 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260319 16:33:46.914464 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.911432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.24.187.195:45293)
0319 16:33:46.911522 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:46.911525 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:46.911533 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:46.911559 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:46.911560 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:46.911571 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:46.911637 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:46.911952 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.912681 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:46.913735 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.914181 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:46.914189 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:46.914211 (+ 22us) server_negotiation.cc:300] Negotiation successful
0319 16:33:46.914247 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":43,"threads_started":1}
I20260319 16:33:46.914556 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.911329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:36825 (local address 127.0.0.1:44792)
0319 16:33:46.911514 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.911522 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:46.911560 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:46.911734 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:46.911735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:46.911736 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:46.911940 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:46.911942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.912599 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:46.912600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:46.913463 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:46.913465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.913527 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:46.913533 (+ 6us) client_negotiation.cc:769] Sending connection context
0319 16:33:46.913560 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:33:46.913595 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260319 16:33:46.914649 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.911234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:45293 (local address 127.0.0.1:34052)
0319 16:33:46.911265 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.911277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:46.911324 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:46.911642 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:46.911643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:46.911645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:46.911871 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:46.911873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.912754 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:46.912755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:46.913566 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:46.913567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.913618 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:46.913620 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:46.913645 (+ 25us) client_negotiation.cc:240] Negotiation successful
0319 16:33:46.913677 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260319 16:33:46.914866 532 tablet_service.cc:1511] Processing CreateTablet for tablet 7c23e52fbe814139beb7104a9ab80c41 (DEFAULT_TABLE table=test-table [id=a1d4ae3acba240b8a41a28fa8f0d7848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:46.914891 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.911305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46733 (local address 127.0.0.1:37344)
0319 16:33:46.911413 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.911421 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:46.911462 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:46.911910 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:46.911911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:46.911913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:46.912098 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:46.912100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.912978 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:46.912979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:46.913594 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:46.913596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:46.913666 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:46.913670 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:46.913697 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:33:46.913754 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":75,"threads_started":1}
I20260319 16:33:46.914891 804 tablet_service.cc:1511] Processing CreateTablet for tablet 7c23e52fbe814139beb7104a9ab80c41 (DEFAULT_TABLE table=test-table [id=a1d4ae3acba240b8a41a28fa8f0d7848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:46.915002 804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c23e52fbe814139beb7104a9ab80c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:46.915004 532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c23e52fbe814139beb7104a9ab80c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:46.915143 668 tablet_service.cc:1511] Processing CreateTablet for tablet 7c23e52fbe814139beb7104a9ab80c41 (DEFAULT_TABLE table=test-table [id=a1d4ae3acba240b8a41a28fa8f0d7848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:46.915256 668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c23e52fbe814139beb7104a9ab80c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:46.916333 952 tablet_bootstrap.cc:492] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc: Bootstrap starting.
I20260319 16:33:46.916501 953 tablet_bootstrap.cc:492] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: Bootstrap starting.
I20260319 16:33:46.916787 952 tablet_bootstrap.cc:654] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:46.916875 953 tablet_bootstrap.cc:654] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:46.917171 954 tablet_bootstrap.cc:492] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Bootstrap starting.
I20260319 16:33:46.917418 953 tablet_bootstrap.cc:492] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: No bootstrap required, opened a new log
I20260319 16:33:46.917426 952 tablet_bootstrap.cc:492] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc: No bootstrap required, opened a new log
I20260319 16:33:46.917459 953 ts_tablet_manager.cc:1403] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:46.917469 952 ts_tablet_manager.cc:1403] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:46.917529 954 tablet_bootstrap.cc:654] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:46.917572 953 raft_consensus.cc:359] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:46.917616 953 raft_consensus.cc:385] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:46.917639 953 raft_consensus.cc:740] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595ada663e79499f80cb25638ccf6ebb, State: Initialized, Role: FOLLOWER
I20260319 16:33:46.917598 952 raft_consensus.cc:359] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:46.917656 952 raft_consensus.cc:385] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:46.917687 952 raft_consensus.cc:740] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d9dd27bba64c15963825d9e53958dc, State: Initialized, Role: FOLLOWER
I20260319 16:33:46.917675 953 consensus_queue.cc:260] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:46.917827 953 ts_tablet_manager.cc:1434] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:46.917830 952 consensus_queue.cc:260] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:46.917966 952 ts_tablet_manager.cc:1434] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:46.918282 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.918030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.193:44699)
0319 16:33:46.918210 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.918245 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260319 16:33:46.918354 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.918168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:43783)
0319 16:33:46.918283 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.918319 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260319 16:33:46.918745 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.918533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:44165)
0319 16:33:46.918672 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.918706 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260319 16:33:46.919003 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.918921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.193:44537)
0319 16:33:46.918953 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.918977 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260319 16:33:46.919283 954 tablet_bootstrap.cc:492] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: No bootstrap required, opened a new log
I20260319 16:33:46.919327 954 ts_tablet_manager.cc:1403] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:46.919461 954 raft_consensus.cc:359] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:46.919510 954 raft_consensus.cc:385] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:46.919523 954 raft_consensus.cc:740] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad8a9f1cb7e420dabb38fe28ae2d633, State: Initialized, Role: FOLLOWER
I20260319 16:33:46.919569 954 consensus_queue.cc:260] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:46.919692 954 ts_tablet_manager.cc:1434] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:46.920070 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.919881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.195:41729)
0319 16:33:46.920000 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.920038 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":72,"threads_started":1}
I20260319 16:33:46.920187 872 heartbeater.cc:499] Master 127.24.187.253:35849 was elected leader, sending a full tablet report...
I20260319 16:33:46.920559 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:46.920392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.195:45283)
0319 16:33:46.920438 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:46.920532 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260319 16:33:47.013314 601 tablet.cc:2379] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:33:47.027571 873 tablet.cc:2379] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:33:47.148414 737 tablet.cc:2379] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:33:47.251356 959 raft_consensus.cc:493] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:47.251466 959 raft_consensus.cc:515] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:47.251801 959 leader_election.cc:290] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195:45293), 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:47.254091 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.252002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:45293 (local address 127.24.187.194:52549)
0319 16:33:47.252047 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.252060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:47.252133 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:47.252298 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:47.252299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:47.252301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:47.252553 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:47.252555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.253211 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:47.253212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:47.253836 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:47.253838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.253896 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:47.253901 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:47.253928 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:33:47.253971 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260319 16:33:47.254103 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.252113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:36825 (local address 127.24.187.194:60575)
0319 16:33:47.252161 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.252172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:47.252223 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:47.252362 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:47.252363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:47.252365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:47.252555 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:47.252557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.253218 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:47.253219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:47.253864 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:47.253867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.253937 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:47.253941 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:47.253965 (+ 24us) client_negotiation.cc:240] Negotiation successful
0319 16:33:47.254008 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260319 16:33:47.254488 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.252129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:52549 (local address 127.24.187.195:45293)
0319 16:33:47.252180 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:47.252182 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:47.252193 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:47.252223 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:47.252225 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:47.252234 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:47.252291 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:47.252645 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.253114 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:47.253914 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.254347 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:47.254355 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:47.254380 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:33:47.254426 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:33:47.254576 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.252203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:60575 (local address 127.24.187.193:36825)
0319 16:33:47.252247 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:47.252249 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:47.252257 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:47.252287 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:47.252288 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:47.252296 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:47.252348 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:47.252642 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.253126 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:47.253981 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.254463 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:47.254472 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:47.254494 (+ 22us) server_negotiation.cc:300] Negotiation successful
0319 16:33:47.254517 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:33:47.254716 824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c23e52fbe814139beb7104a9ab80c41" candidate_uuid: "58d9dd27bba64c15963825d9e53958dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" is_pre_election: true
I20260319 16:33:47.254776 552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c23e52fbe814139beb7104a9ab80c41" candidate_uuid: "58d9dd27bba64c15963825d9e53958dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595ada663e79499f80cb25638ccf6ebb" is_pre_election: true
I20260319 16:33:47.254837 824 raft_consensus.cc:2468] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d9dd27bba64c15963825d9e53958dc in term 0.
I20260319 16:33:47.254845 552 raft_consensus.cc:2468] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d9dd27bba64c15963825d9e53958dc in term 0.
I20260319 16:33:47.255030 621 leader_election.cc:304] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d9dd27bba64c15963825d9e53958dc, 595ada663e79499f80cb25638ccf6ebb; no voters:
I20260319 16:33:47.255187 959 raft_consensus.cc:2804] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:47.255245 959 raft_consensus.cc:493] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:47.255261 959 raft_consensus.cc:3060] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:47.255960 959 raft_consensus.cc:515] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:47.256083 959 leader_election.cc:290] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [CANDIDATE]: Term 1 election: Requested vote from peers 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195:45293), 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825)
I20260319 16:33:47.256313 552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c23e52fbe814139beb7104a9ab80c41" candidate_uuid: "58d9dd27bba64c15963825d9e53958dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595ada663e79499f80cb25638ccf6ebb"
I20260319 16:33:47.256317 824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c23e52fbe814139beb7104a9ab80c41" candidate_uuid: "58d9dd27bba64c15963825d9e53958dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633"
I20260319 16:33:47.256382 552 raft_consensus.cc:3060] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:47.256383 824 raft_consensus.cc:3060] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:47.256963 552 raft_consensus.cc:2468] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d9dd27bba64c15963825d9e53958dc in term 1.
I20260319 16:33:47.257093 824 raft_consensus.cc:2468] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d9dd27bba64c15963825d9e53958dc in term 1.
I20260319 16:33:47.257148 621 leader_election.cc:304] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d9dd27bba64c15963825d9e53958dc, 595ada663e79499f80cb25638ccf6ebb; no voters:
I20260319 16:33:47.257249 959 raft_consensus.cc:2804] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:47.257392 959 raft_consensus.cc:697] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 1 LEADER]: Becoming Leader. State: Replica: 58d9dd27bba64c15963825d9e53958dc, State: Running, Role: LEADER
I20260319 16:33:47.257467 959 consensus_queue.cc:237] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } }
I20260319 16:33:47.257750 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.257598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:41987)
0319 16:33:47.257639 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.257696 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260319 16:33:47.257475 736 heartbeater.cc:499] Master 127.24.187.253:35849 was elected leader, sending a full tablet report...
I20260319 16:33:47.258250 32688 catalog_manager.cc:5671] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d9dd27bba64c15963825d9e53958dc (127.24.187.194). New cstate: current_term: 1 leader_uuid: "58d9dd27bba64c15963825d9e53958dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d9dd27bba64c15963825d9e53958dc" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46733 } health_report { overall_health: HEALTHY } } }
I20260319 16:33:47.267148 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.265164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.24.187.194:46733)
0319 16:33:47.265217 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:47.265219 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:47.265226 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:33:47.265260 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:47.265260 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:47.265276 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:47.265323 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:47.265567 (+ 244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.266076 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:47.266707 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:47.266840 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:47.266871 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:47.266965 (+ 94us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:47.267023 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:47.267073 (+ 50us) server_negotiation.cc:300] Negotiation successful
0319 16:33:47.267094 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:33:47.268262 552 raft_consensus.cc:1275] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Refusing update from remote peer 58d9dd27bba64c15963825d9e53958dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:47.268353 824 raft_consensus.cc:1275] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 FOLLOWER]: Refusing update from remote peer 58d9dd27bba64c15963825d9e53958dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:47.268570 959 consensus_queue.cc:1048] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "595ada663e79499f80cb25638ccf6ebb" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:47.268709 965 consensus_queue.cc:1048] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad8a9f1cb7e420dabb38fe28ae2d633" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:47.269169 600 heartbeater.cc:499] Master 127.24.187.253:35849 was elected leader, sending a full tablet report...
I20260319 16:33:47.269467 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.269222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.193:40031)
0319 16:33:47.269398 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.269437 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260319 16:33:47.269591 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.269486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.195:44379)
0319 16:33:47.269531 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.269561 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260319 16:33:47.270888 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.270764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:53181)
0319 16:33:47.270833 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.270857 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260319 16:33:47.271202 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.271111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.193:36305)
0319 16:33:47.271148 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.271172 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260319 16:33:47.271564 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.271475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.195:60179)
0319 16:33:47.271510 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.271534 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260319 16:33:47.272359 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.272233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:45081 (local address 127.24.187.194:57677)
0319 16:33:47.272305 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.272333 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:45081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260319 16:33:47.272761 32688 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:47.272847 32688 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:47.273979 32688 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 919c26ab3f984ce6a3546c6ce1131317: Sending DeleteTablet for 3 replicas of tablet 7c23e52fbe814139beb7104a9ab80c41
I20260319 16:33:47.274223 668 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c23e52fbe814139beb7104a9ab80c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:47 UTC) from {username='oryx', principal='oryx/127.24.187.253@KRBTEST.COM'} at 127.0.0.1:37344
I20260319 16:33:47.274331 804 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c23e52fbe814139beb7104a9ab80c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:47 UTC) from {username='oryx', principal='oryx/127.24.187.253@KRBTEST.COM'} at 127.0.0.1:34052
I20260319 16:33:47.274379 532 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c23e52fbe814139beb7104a9ab80c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:47 UTC) from {username='oryx', principal='oryx/127.24.187.253@KRBTEST.COM'} at 127.0.0.1:44792
I20260319 16:33:47.274528 979 tablet_replica.cc:333] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: stopping tablet replica
I20260319 16:33:47.274538 980 tablet_replica.cc:333] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: stopping tablet replica
I20260319 16:33:47.274590 980 raft_consensus.cc:2243] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:47.274590 979 raft_consensus.cc:2243] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:47.274631 980 raft_consensus.cc:2272] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:47.274636 979 raft_consensus.cc:2272] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:47.274962 980 ts_tablet_manager.cc:1916] T 7c23e52fbe814139beb7104a9ab80c41 P 595ada663e79499f80cb25638ccf6ebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:47.274962 979 ts_tablet_manager.cc:1916] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:47.275133 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 469
I20260319 16:33:47.276927 979 ts_tablet_manager.cc:1929] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:47.276979 979 log.cc:1199] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/7c23e52fbe814139beb7104a9ab80c41
I20260319 16:33:47.277225 979 ts_tablet_manager.cc:1950] T 7c23e52fbe814139beb7104a9ab80c41 P 3ad8a9f1cb7e420dabb38fe28ae2d633: Deleting consensus metadata
I20260319 16:33:47.277546 32672 catalog_manager.cc:5002] TS 3ad8a9f1cb7e420dabb38fe28ae2d633 (127.24.187.195:45293): tablet 7c23e52fbe814139beb7104a9ab80c41 (table test-table [id=a1d4ae3acba240b8a41a28fa8f0d7848]) successfully deleted
W20260319 16:33:47.282096 947 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:47.281944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:36825 (local address 127.0.0.1:44802)
0319 16:33:47.282029 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.282049 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:36825: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":67}
I20260319 16:33:47.282187 978 tablet_replica.cc:333] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc: stopping tablet replica
I20260319 16:33:47.282258 978 raft_consensus.cc:2243] T 7c23e52fbe814139beb7104a9ab80c41 P 58d9dd27bba64c15963825d9e53958dc [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:47.282277 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 605
W20260319 16:33:47.283420 32672 catalog_manager.cc:4729] TS 595ada663e79499f80cb25638ccf6ebb (127.24.187.193:36825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c23e52fbe814139beb7104a9ab80c41: Network error: Client connection negotiation failed: client connection to 127.24.187.193:36825: connect: Connection reset by peer (error 104)
I20260319 16:33:47.286942 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 741
I20260319 16:33:47.287032 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:47.286915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46733 (local address 127.0.0.1:37356)
0319 16:33:47.286971 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:47.287001 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.194:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260319 16:33:47.292471 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 32553
I20260319 16:33:47.296651 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 32744
2026-03-19T16:33:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6670 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:47 dist-test-slave-svn8 krb5kdc[983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:47 dist-test-slave-svn8 krb5kdc[983](info): set up 2 sockets
Mar 19 16:33:47 dist-test-slave-svn8 krb5kdc[983](info): commencing operation
krb5kdc: starting...
W20260319 16:33:49.130950 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.813s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:49.269393 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:46305
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:46305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:49.342865 999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:49.343027 999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:49.343048 999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:49.344533 999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:49.344583 999 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260319 16:33:49.344595 999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:49.344606 999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:49.344617 999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:49.344628 999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:49.346295 999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:46305
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:46305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:49.346510 999 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:49.346689 999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:49.349013 1007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:49.349085 999 server_base.cc:1060] running on GCE node
W20260319 16:33:49.349042 1005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:49.349013 1004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:49.349375 999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:49.349623 999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:49.350759 999 hybrid_clock.cc:648] HybridClock initialized: now 1773938029350737 us; error 31 us; skew 500 ppm
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:49.353040 999 init.cc:377] Logged in from keytab as oryx/127.24.187.254@KRBTEST.COM (short username oryx)
I20260319 16:33:49.353513 999 webserver.cc:492] Webserver started at http://127.24.187.254:41689/ using document root <none> and password file <none>
I20260319 16:33:49.353657 999 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:49.353693 999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:49.353806 999 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:49.354701 999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "268cc07ed887430cacdbb244cdd1ad75"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "8eaf4e61c2d32a3b697a78ab80e12180"
server_key_iv: "56d053893aa3bfaad7720246e285482b"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.354956 999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "268cc07ed887430cacdbb244cdd1ad75"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "8eaf4e61c2d32a3b697a78ab80e12180"
server_key_iv: "56d053893aa3bfaad7720246e285482b"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.355967 999 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:49.356528 1014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.356693 999 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:49.356750 999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "268cc07ed887430cacdbb244cdd1ad75"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "8eaf4e61c2d32a3b697a78ab80e12180"
server_key_iv: "56d053893aa3bfaad7720246e285482b"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.356810 999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:49.370386 999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:49.372272 999 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:49.372395 999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:49.376765 999 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:46305
I20260319 16:33:49.376809 1076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:46305 every 8 connection(s)
I20260319 16:33:49.377095 999 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:49.377627 1077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:49.380113 1077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Bootstrap starting.
I20260319 16:33:49.380662 1077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:49.380919 1077 log.cc:826] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:49.381561 1077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: No bootstrap required, opened a new log
I20260319 16:33:49.382767 1077 raft_consensus.cc:359] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cc07ed887430cacdbb244cdd1ad75" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46305 } }
I20260319 16:33:49.382885 1077 raft_consensus.cc:385] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:49.382907 1077 raft_consensus.cc:740] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268cc07ed887430cacdbb244cdd1ad75, State: Initialized, Role: FOLLOWER
I20260319 16:33:49.383018 1077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cc07ed887430cacdbb244cdd1ad75" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46305 } }
I20260319 16:33:49.383080 1077 raft_consensus.cc:399] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:49.383101 1077 raft_consensus.cc:493] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:49.383143 1077 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:49.383544 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 999
I20260319 16:33:49.383638 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:49.383795 25327 external_mini_cluster.cc:1468] Setting key a485644be8f9001143505281aacb0baa
I20260319 16:33:49.383795 1077 raft_consensus.cc:515] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cc07ed887430cacdbb244cdd1ad75" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46305 } }
I20260319 16:33:49.383893 1077 leader_election.cc:304] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268cc07ed887430cacdbb244cdd1ad75; no voters:
I20260319 16:33:49.384037 1077 leader_election.cc:290] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:49.384086 1080 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:49.384230 1080 raft_consensus.cc:697] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [term 1 LEADER]: Becoming Leader. State: Replica: 268cc07ed887430cacdbb244cdd1ad75, State: Running, Role: LEADER
I20260319 16:33:49.384265 1077 sys_catalog.cc:565] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:49.384332 1080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cc07ed887430cacdbb244cdd1ad75" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46305 } }
I20260319 16:33:49.384821 1081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "268cc07ed887430cacdbb244cdd1ad75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cc07ed887430cacdbb244cdd1ad75" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46305 } } }
I20260319 16:33:49.384891 1081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:49.384918 1082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268cc07ed887430cacdbb244cdd1ad75. Latest consensus state: current_term: 1 leader_uuid: "268cc07ed887430cacdbb244cdd1ad75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268cc07ed887430cacdbb244cdd1ad75" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46305 } } }
I20260319 16:33:49.384995 1082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:49.385136 1091 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:49.385565 1091 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:49.387266 1091 catalog_manager.cc:1357] Generated new cluster ID: 0896d1b55ff24546b1c9d21f7d75e016
I20260319 16:33:49.387322 1091 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:49.393894 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.384441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.24.187.254:46305)
0319 16:33:49.384580 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.384592 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.384804 (+ 212us) server_negotiation.cc:408] Connection header received
0319 16:33:49.384947 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.384954 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.385002 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.385117 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.386300 (+ 1183us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.387189 (+ 889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.387953 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.388200 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.390544 (+ 2344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.390569 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.390570 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.390591 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.392500 (+ 1909us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.392873 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.392875 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.392876 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.392942 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.393084 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.393084 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.393085 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.393316 (+ 231us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.393446 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.393572 (+ 126us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.393695 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260319 16:33:49.405979 1091 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:49.406816 1091 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:49.427979 1091 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Generated new TSK 0
I20260319 16:33:49.428205 1091 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:49.484514 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:46305
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:49.558670 1105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:49.558830 1105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:49.558852 1105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:49.560467 1105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:49.560508 1105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:49.560527 1105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:33:49.560544 1105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:49.562161 1105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:49.562328 1105 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:49.562453 1105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:49.564744 1113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:49.564831 1111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:49.564740 1110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:49.564855 1105 server_base.cc:1060] running on GCE node
I20260319 16:33:49.565102 1105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:49.565284 1105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:49.566412 1105 hybrid_clock.cc:648] HybridClock initialized: now 1773938029566390 us; error 28 us; skew 500 ppm
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:49.568737 1105 init.cc:377] Logged in from keytab as oryx/127.24.187.193@KRBTEST.COM (short username oryx)
I20260319 16:33:49.569193 1105 webserver.cc:492] Webserver started at http://127.24.187.193:37703/ using document root <none> and password file <none>
I20260319 16:33:49.569343 1105 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:49.569380 1105 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:49.569478 1105 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:49.570382 1105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "7cefd550566a43b7853389216dfa2655"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "e523de71dd2888de98f2f04986c3b487"
server_key_iv: "6f4c7a4621c003bfaf1474eeb8426b45"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.570674 1105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "7cefd550566a43b7853389216dfa2655"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "e523de71dd2888de98f2f04986c3b487"
server_key_iv: "6f4c7a4621c003bfaf1474eeb8426b45"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.571722 1105 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:33:49.572311 1120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.572496 1105 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:49.572561 1105 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "7cefd550566a43b7853389216dfa2655"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "e523de71dd2888de98f2f04986c3b487"
server_key_iv: "6f4c7a4621c003bfaf1474eeb8426b45"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.572623 1105 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:49.588995 1105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:49.590691 1105 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:49.590811 1105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:49.591303 1105 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:49.591341 1105 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.591369 1105 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:49.591385 1105 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:49.598043 1105 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:39655
I20260319 16:33:49.598121 1235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:39655 every 8 connection(s)
I20260319 16:33:49.598428 1105 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:33:49.598973 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 1105
I20260319 16:33:49.599063 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:49.599258 25327 external_mini_cluster.cc:1468] Setting key cf09f45bf702a2f4b2d8da63ace99ead
I20260319 16:33:49.600101 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.591692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:35985 (local address 127.24.187.254:46305)
0319 16:33:49.591761 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.591764 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.591895 (+ 131us) server_negotiation.cc:408] Connection header received
0319 16:33:49.592384 (+ 489us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.592386 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.592403 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.592480 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.593422 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.594201 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.594959 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.595141 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.597116 (+ 1975us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.597121 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.597122 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.597131 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.598740 (+ 1609us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.599183 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.599184 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.599185 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.599208 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.599450 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.599452 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.599452 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.599581 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.599668 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.599905 (+ 237us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.600011 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260319 16:33:49.600173 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.591625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46305 (local address 127.24.187.193:35985)
0319 16:33:49.591786 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.591816 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.592295 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.592534 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.592538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.592884 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:49.593305 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.593312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.594291 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.594292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.594874 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.594876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.595004 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.595453 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:49.595478 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:49.597035 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:49.598835 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.598839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.598841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.599105 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.599274 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.599274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.599275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.599352 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.599660 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:49.599666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:49.599786 (+ 120us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.599879 (+ 93us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.600004 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":106,"threads_started":1}
I20260319 16:33:49.600756 1236 heartbeater.cc:344] Connected to a master server at 127.24.187.254:46305
I20260319 16:33:49.600836 1236 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:49.601060 1236 heartbeater.cc:507] Master 127.24.187.254:46305 requested a full tablet report, sending...
I20260319 16:33:49.601594 1030 ts_manager.cc:194] Registered new tserver with Master: 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655)
I20260319 16:33:49.602515 1030 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:35985
I20260319 16:33:49.606496 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.601966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:39517 (local address 127.24.187.254:46305)
0319 16:33:49.602006 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.602007 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.602119 (+ 112us) server_negotiation.cc:408] Connection header received
0319 16:33:49.602174 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.602175 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.602182 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.602221 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.602732 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.603223 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.603734 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.603890 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.604492 (+ 602us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.604495 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.604495 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.604501 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.605652 (+ 1151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.605934 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.605935 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.605935 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.605955 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.606080 (+ 125us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.606080 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.606081 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.606158 (+ 77us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.606209 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.606334 (+ 125us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.606404 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:33:49.606509 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.601888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46305 (local address 127.24.187.193:39517)
0319 16:33:49.602048 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.602059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.602123 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.602234 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.602235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.602384 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:49.602605 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.602607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.603297 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.603298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.603661 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.603663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.603729 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.604163 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:49.604167 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:49.604388 (+ 221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:49.605743 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.605744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.605745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.605878 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.606002 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.606003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.606003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.606032 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.606205 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:49.606206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:49.606248 (+ 42us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.606309 (+ 61us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.606403 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:49.648432 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:46305
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:49.723809 1247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:49.724083 1247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:49.724223 1247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:49.725862 1247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:49.725909 1247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:49.725930 1247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:33:49.725952 1247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:49.727635 1247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:49.727818 1247 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:49.727985 1247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:33:49.730710 1247 server_base.cc:1060] running on GCE node
W20260319 16:33:49.730682 1252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:49.730675 1255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:49.730662 1253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:49.731098 1247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:49.731304 1247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:49.732473 1247 hybrid_clock.cc:648] HybridClock initialized: now 1773938029732471 us; error 43 us; skew 500 ppm
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:49.735098 1247 init.cc:377] Logged in from keytab as oryx/127.24.187.194@KRBTEST.COM (short username oryx)
I20260319 16:33:49.735666 1247 webserver.cc:492] Webserver started at http://127.24.187.194:33495/ using document root <none> and password file <none>
I20260319 16:33:49.735833 1247 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:49.735868 1247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:49.735967 1247 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:49.736886 1247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "68266dcfdb2e440da4de02e54e829c8d"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "c1741cea0850c790df1633c46a10ef08"
server_key_iv: "2666f1de84a5a7088cdb734bbe6abad4"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.737192 1247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "68266dcfdb2e440da4de02e54e829c8d"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "c1741cea0850c790df1633c46a10ef08"
server_key_iv: "2666f1de84a5a7088cdb734bbe6abad4"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.738469 1247 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:33:49.739161 1262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.739349 1247 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:49.739450 1247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "68266dcfdb2e440da4de02e54e829c8d"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "c1741cea0850c790df1633c46a10ef08"
server_key_iv: "2666f1de84a5a7088cdb734bbe6abad4"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.739513 1247 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:49.762914 1247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:49.764838 1247 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:49.764969 1247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:49.765689 1247 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:49.765794 1247 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.765841 1247 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:49.765865 1247 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.771792 1247 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:43279
I20260319 16:33:49.771956 1377 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:43279 every 8 connection(s)
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:49.772125 1247 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:49.773236 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 1247
I20260319 16:33:49.773339 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:49.773527 25327 external_mini_cluster.cc:1468] Setting key eb5e36c0227aedbaf53c19ee403ac522
I20260319 16:33:49.775132 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.765968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:54909 (local address 127.24.187.254:46305)
0319 16:33:49.766020 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.766022 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.766210 (+ 188us) server_negotiation.cc:408] Connection header received
0319 16:33:49.767005 (+ 795us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.767007 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.767021 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.767101 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.768088 (+ 987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.768829 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.769602 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.769808 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.772566 (+ 2758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.772570 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.772570 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.772577 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.774006 (+ 1429us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.774388 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.774389 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.774389 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.774413 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.774580 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.774580 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.774581 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.774675 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.774728 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.774959 (+ 231us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.775047 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:33:49.775229 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.765953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46305 (local address 127.24.187.194:54909)
0319 16:33:49.766099 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.766128 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.766891 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.767164 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.767169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.767589 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:49.767982 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.767986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.768927 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.768927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.769517 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.769519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.769640 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.770183 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:49.770201 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:49.772485 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:49.774076 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.774081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.774083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.774325 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.774463 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.774464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.774464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.774532 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.774723 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:49.774727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:49.774847 (+ 120us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.774935 (+ 88us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.775064 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260319 16:33:49.775738 1378 heartbeater.cc:344] Connected to a master server at 127.24.187.254:46305
I20260319 16:33:49.775832 1378 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:49.776033 1378 heartbeater.cc:507] Master 127.24.187.254:46305 requested a full tablet report, sending...
I20260319 16:33:49.776417 1030 ts_manager.cc:194] Registered new tserver with Master: 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279)
I20260319 16:33:49.776924 1030 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:54909
WARNING: no policy specified for oryx/127.24.187.195@KRBTEST.COM; defaulting to no policy
I20260319 16:33:49.782455 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.776958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:57939 (local address 127.24.187.254:46305)
0319 16:33:49.776993 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.776995 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.777166 (+ 171us) server_negotiation.cc:408] Connection header received
0319 16:33:49.777618 (+ 452us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.777619 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.777630 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.777685 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.778083 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.778793 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.779323 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.779491 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.780045 (+ 554us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.780048 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.780048 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.780054 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.781346 (+ 1292us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.781645 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.781646 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.781647 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.781671 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.781876 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.781877 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.781877 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.782016 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.782080 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.782288 (+ 208us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.782376 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:33:49.782464 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.776900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46305 (local address 127.24.187.194:57939)
0319 16:33:49.777096 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.777106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.777533 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.777700 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.777701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.777849 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:49.778015 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.778016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.778881 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.778883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.779250 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.779252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.779310 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.779738 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:49.779742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:49.779969 (+ 227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:49.781449 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.781450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.781450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.781580 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.781790 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.781791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.781791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.781820 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.782109 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:49.782111 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:49.782160 (+ 49us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.782232 (+ 72us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.782344 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab.
I20260319 16:33:49.825254 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:46305
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:49.899091 1389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:49.899240 1389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:49.899261 1389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:49.900707 1389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:49.900744 1389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:49.900763 1389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260319 16:33:49.900779 1389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:49.902459 1389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/krb5kdc/oryx.keytab
--principal=oryx/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.187.254:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:49.902647 1389 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:49.902807 1389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:49.905022 1395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:49.905022 1396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:49.905078 1398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:49.905333 1389 server_base.cc:1060] running on GCE node
I20260319 16:33:49.905488 1389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:49.905673 1389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:49.906842 1389 hybrid_clock.cc:648] HybridClock initialized: now 1773938029906826 us; error 31 us; skew 500 ppm
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:49.909313 1389 init.cc:377] Logged in from keytab as oryx/127.24.187.195@KRBTEST.COM (short username oryx)
I20260319 16:33:49.909891 1389 webserver.cc:492] Webserver started at http://127.24.187.195:35575/ using document root <none> and password file <none>
I20260319 16:33:49.910059 1389 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:49.910094 1389 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:49.910251 1389 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:49.911244 1389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "6ccbfff35ffa4b63b510e428d594920f"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "b07eba56a23706fcaf60ce360f4bd94c"
server_key_iv: "dc6d57a980fa7be0fb02cc7baacf18fc"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.911580 1389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "6ccbfff35ffa4b63b510e428d594920f"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "b07eba56a23706fcaf60ce360f4bd94c"
server_key_iv: "dc6d57a980fa7be0fb02cc7baacf18fc"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.912690 1389 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260319 16:33:49.913300 1405 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.913497 1389 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:49.913556 1389 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "6ccbfff35ffa4b63b510e428d594920f"
format_stamp: "Formatted at 2026-03-19 16:33:49 on dist-test-slave-svn8"
server_key: "b07eba56a23706fcaf60ce360f4bd94c"
server_key_iv: "dc6d57a980fa7be0fb02cc7baacf18fc"
server_key_version: "encryptionkey@0"
I20260319 16:33:49.913619 1389 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:49.926674 1389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:49.928453 1389 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:49.928561 1389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:49.929010 1389 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:49.929045 1389 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:49.929072 1389 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:49.929095 1389 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:49.935662 1389 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:42561
I20260319 16:33:49.935714 1521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:42561 every 8 connection(s)
I20260319 16:33:49.936034 1389 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:33:49.936684 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.929288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:52339 (local address 127.24.187.254:46305)
0319 16:33:49.929361 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.929364 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.929579 (+ 215us) server_negotiation.cc:408] Connection header received
0319 16:33:49.930131 (+ 552us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.930134 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.930147 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.930226 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.931052 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.931518 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.932034 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.932198 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.934000 (+ 1802us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.934004 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.934005 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.934012 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.935516 (+ 1504us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.935927 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.935928 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.935929 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.935956 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.936120 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.936121 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.936122 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.936211 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.936273 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.936518 (+ 245us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.936597 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260319 16:33:49.936776 1414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.929295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46305 (local address 127.24.187.195:52339)
0319 16:33:49.929461 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.929492 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.930037 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.930293 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.930297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.930668 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:49.930964 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.930968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.931590 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.931591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.931961 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.931963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.932068 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.932562 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:49.932581 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:49.933922 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:49.935597 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.935600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.935602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.935855 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.936015 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.936016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.936016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.936064 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.936263 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:49.936267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:49.936396 (+ 129us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.936499 (+ 103us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.936612 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260319 16:33:49.937170 1522 heartbeater.cc:344] Connected to a master server at 127.24.187.254:46305
I20260319 16:33:49.937242 1522 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:49.937398 1522 heartbeater.cc:507] Master 127.24.187.254:46305 requested a full tablet report, sending...
I20260319 16:33:49.937774 1031 ts_manager.cc:194] Registered new tserver with Master: 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:49.938196 1031 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:52339
I20260319 16:33:49.939574 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 1389
I20260319 16:33:49.939661 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:49.939795 25327 external_mini_cluster.cc:1468] Setting key 9a54907c881d2cd6854ae41c2561f366
I20260319 16:33:49.940654 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:33:49.942600 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.938075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46305 (local address 127.24.187.195:40873)
0319 16:33:49.938193 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.938205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.938269 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.938399 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.938400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.938549 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:49.938711 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.938713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.939295 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.939295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.939646 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.939647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.939710 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.940083 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:49.940088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:49.940376 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:49.941858 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.941859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.941859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.941998 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.942130 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:49.942130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:49.942130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:49.942152 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:49.942319 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:49.942320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:49.942387 (+ 67us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.942454 (+ 67us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.942527 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260319 16:33:49.942631 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.938148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:40873 (local address 127.24.187.254:46305)
0319 16:33:49.938187 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.938189 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.938254 (+ 65us) server_negotiation.cc:408] Connection header received
0319 16:33:49.938319 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.938320 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.938329 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.938385 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.938787 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.939235 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.939762 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.939910 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.940455 (+ 545us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.940457 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.940458 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.940463 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.941787 (+ 1324us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.942059 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.942060 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.942060 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.942082 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.942188 (+ 106us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.942188 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.942188 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.942276 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.942326 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.942485 (+ 159us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.942552 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
W20260319 16:33:49.960831 1084 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:49.955818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.24.187.254:46305)
0319 16:33:49.955884 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.955886 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.955895 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:49.955921 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.955922 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.955931 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.955980 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.956639 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.957121 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.957796 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.957930 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.960772 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48236: BlockingRecv error: recv got EOF from 127.0.0.1:48236 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Mar 19 16:33:49 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:49.967665 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.961689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.24.187.254:46305)
0319 16:33:49.961770 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.961772 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.961830 (+ 58us) server_negotiation.cc:408] Connection header received
0319 16:33:49.961890 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.961892 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.961901 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.961946 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:49.962485 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.962928 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.963409 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.963573 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.965338 (+ 1765us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:49.965342 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:49.965342 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:49.965349 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:49.966634 (+ 1285us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.966919 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.966920 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.966920 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.966941 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:49.967207 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:49.967208 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:49.967208 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:49.967303 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:49.967395 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.967495 (+ 100us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.967575 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260319 16:33:49.968992 1031 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:49.970275 1031 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:49.975600 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.972512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42561 (local address 127.0.0.1:56220)
0319 16:33:49.972699 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.972708 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.972752 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.973028 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.973029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.973030 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:49.973221 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.973223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.974570 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.974572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.975295 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.975298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.975389 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.975436 (+ 47us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.975482 (+ 46us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.975523 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260319 16:33:49.976006 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.972317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43279 (local address 127.0.0.1:51066)
0319 16:33:49.972500 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.972530 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.972607 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.972814 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.972817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.972820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:49.973078 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.973080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.974547 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.974548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.975445 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.975446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.975502 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.975507 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.975538 (+ 31us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.975576 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260319 16:33:49.976133 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.972616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.24.187.195:42561)
0319 16:33:49.972871 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.972878 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.972888 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:49.972923 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.972925 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.972941 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.973012 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:49.973301 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.974487 (+ 1186us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.975420 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.975929 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.975984 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.976012 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.976053 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":108,"threads_started":1}
I20260319 16:33:49.976339 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.972686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:39655 (local address 127.0.0.1:38352)
0319 16:33:49.972812 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:49.972821 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:49.972868 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:49.973415 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:49.973416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:49.973417 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:49.973570 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.973571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.974484 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.974486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:49.975430 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:49.975433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.975505 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.975508 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:49.975538 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:33:49.975576 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
I20260319 16:33:49.976548 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.972901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.24.187.193:39655)
0319 16:33:49.973146 (+ 245us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.973153 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.973163 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:49.973214 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.973216 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.973231 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.973290 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:49.973647 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.974395 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.975642 (+ 1247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.976157 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.976205 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.976237 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.976474 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260319 16:33:49.976744 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 9608adfe31074b108e6c042dc4e429d2 (DEFAULT_TABLE table=test-table [id=fc2068564a7b4be09ca90cde275450df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:49.977048 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9608adfe31074b108e6c042dc4e429d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:49.977272 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:49.972420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.24.187.194:43279)
0319 16:33:49.972613 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:49.972620 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:49.972633 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:33:49.972690 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:49.972693 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:49.972710 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:49.973174 (+ 464us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:49.973294 (+ 120us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.974468 (+ 1174us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:49.976480 (+ 2012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:49.977091 (+ 611us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:49.977134 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:49.977165 (+ 31us) server_negotiation.cc:300] Negotiation successful
0319 16:33:49.977204 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":114,"threads_started":1}
I20260319 16:33:49.977259 1170 tablet_service.cc:1511] Processing CreateTablet for tablet 9608adfe31074b108e6c042dc4e429d2 (DEFAULT_TABLE table=test-table [id=fc2068564a7b4be09ca90cde275450df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:49.977520 1170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9608adfe31074b108e6c042dc4e429d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:49.979897 1550 tablet_bootstrap.cc:492] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Bootstrap starting.
I20260319 16:33:49.980048 1551 tablet_bootstrap.cc:492] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Bootstrap starting.
I20260319 16:33:49.980222 1312 tablet_service.cc:1511] Processing CreateTablet for tablet 9608adfe31074b108e6c042dc4e429d2 (DEFAULT_TABLE table=test-table [id=fc2068564a7b4be09ca90cde275450df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:49.980451 1312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9608adfe31074b108e6c042dc4e429d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:49.980684 1551 tablet_bootstrap.cc:654] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:49.980700 1550 tablet_bootstrap.cc:654] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:49.981009 1550 log.cc:826] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:49.981544 1551 log.cc:826] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:49.982154 1550 tablet_bootstrap.cc:492] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: No bootstrap required, opened a new log
I20260319 16:33:49.982168 1551 tablet_bootstrap.cc:492] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: No bootstrap required, opened a new log
I20260319 16:33:49.982224 1551 ts_tablet_manager.cc:1403] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260319 16:33:49.982224 1550 ts_tablet_manager.cc:1403] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:49.983026 1554 tablet_bootstrap.cc:492] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Bootstrap starting.
I20260319 16:33:49.983596 1550 raft_consensus.cc:359] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:49.983564 1551 raft_consensus.cc:359] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:49.983697 1554 tablet_bootstrap.cc:654] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:49.983707 1550 raft_consensus.cc:385] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:49.983708 1551 raft_consensus.cc:385] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:49.983736 1550 raft_consensus.cc:740] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccbfff35ffa4b63b510e428d594920f, State: Initialized, Role: FOLLOWER
I20260319 16:33:49.983739 1551 raft_consensus.cc:740] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Initialized, Role: FOLLOWER
I20260319 16:33:49.983819 1550 consensus_queue.cc:260] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:49.983856 1551 consensus_queue.cc:260] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:49.983982 1554 log.cc:826] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:49.984055 1550 ts_tablet_manager.cc:1434] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:49.984118 1551 ts_tablet_manager.cc:1434] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:33:49.984148 1236 heartbeater.cc:499] Master 127.24.187.254:46305 was elected leader, sending a full tablet report...
I20260319 16:33:49.984196 1522 heartbeater.cc:499] Master 127.24.187.254:46305 was elected leader, sending a full tablet report...
I20260319 16:33:49.985200 1554 tablet_bootstrap.cc:492] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: No bootstrap required, opened a new log
I20260319 16:33:49.985256 1554 ts_tablet_manager.cc:1403] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:49.986404 1554 raft_consensus.cc:359] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:49.986526 1554 raft_consensus.cc:385] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:49.986557 1554 raft_consensus.cc:740] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68266dcfdb2e440da4de02e54e829c8d, State: Initialized, Role: FOLLOWER
I20260319 16:33:49.986654 1554 consensus_queue.cc:260] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:49.986886 1554 ts_tablet_manager.cc:1434] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:33:49.986958 1378 heartbeater.cc:499] Master 127.24.187.254:46305 was elected leader, sending a full tablet report...
W20260319 16:33:50.022637 1379 tablet.cc:2379] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:33:50.099126 1237 tablet.cc:2379] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:33:50.109231 1556 raft_consensus.cc:493] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:50.109349 1556 raft_consensus.cc:515] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:50.109673 1556 leader_election.cc:290] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:50.111977 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.109859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43279 (local address 127.24.187.193:60887)
0319 16:33:50.109983 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:50.109996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:50.110062 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:50.110257 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:50.110257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:50.110259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:50.110411 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.110412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.111143 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.111144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:50.111805 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.111808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.111865 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.111869 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:50.111889 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:33:50.111915 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":53,"threads_started":1}
I20260319 16:33:50.112187 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.110017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42561 (local address 127.24.187.193:34335)
0319 16:33:50.110115 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:50.110127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:50.110209 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:50.110340 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:50.110341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:50.110342 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:50.110579 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.110608 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.111340 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.111342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:50.111939 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.111942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.112028 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.112032 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:50.112070 (+ 38us) client_negotiation.cc:240] Negotiation successful
0319 16:33:50.112106 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260319 16:33:50.112468 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.109952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:60887 (local address 127.24.187.194:43279)
0319 16:33:50.110010 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.110012 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.110045 (+ 33us) server_negotiation.cc:408] Connection header received
0319 16:33:50.110159 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.110161 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.110170 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.110194 (+ 24us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:50.110474 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.111051 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.111899 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.112345 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.112355 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.112381 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.112416 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:33:50.112571 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.110081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:34335 (local address 127.24.187.195:42561)
0319 16:33:50.110122 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.110124 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.110195 (+ 71us) server_negotiation.cc:408] Connection header received
0319 16:33:50.110272 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.110273 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.110283 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.110337 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:50.110710 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.111235 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.112037 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.112467 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.112474 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.112493 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.112514 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:33:50.112792 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9608adfe31074b108e6c042dc4e429d2" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d" is_pre_election: true
I20260319 16:33:50.112850 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9608adfe31074b108e6c042dc4e429d2" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f" is_pre_election: true
I20260319 16:33:50.112947 1475 raft_consensus.cc:2468] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 0.
I20260319 16:33:50.112947 1332 raft_consensus.cc:2468] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 0.
I20260319 16:33:50.113138 1122 leader_election.cc:304] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ccbfff35ffa4b63b510e428d594920f, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:50.113299 1556 raft_consensus.cc:2804] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:50.113363 1556 raft_consensus.cc:493] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:50.113385 1556 raft_consensus.cc:3060] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.114084 1556 raft_consensus.cc:515] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:50.114198 1556 leader_election.cc:290] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 election: Requested vote from peers 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:50.114375 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9608adfe31074b108e6c042dc4e429d2" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d"
I20260319 16:33:50.114395 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9608adfe31074b108e6c042dc4e429d2" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f"
I20260319 16:33:50.114444 1332 raft_consensus.cc:3060] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.114471 1475 raft_consensus.cc:3060] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.115064 1332 raft_consensus.cc:2468] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 1.
I20260319 16:33:50.115187 1475 raft_consensus.cc:2468] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 1.
I20260319 16:33:50.115227 1122 leader_election.cc:304] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68266dcfdb2e440da4de02e54e829c8d, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:50.115299 1556 raft_consensus.cc:2804] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:50.115389 1556 raft_consensus.cc:697] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 1 LEADER]: Becoming Leader. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Running, Role: LEADER
I20260319 16:33:50.115482 1556 consensus_queue.cc:237] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } }
I20260319 16:33:50.116266 1031 catalog_manager.cc:5671] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cefd550566a43b7853389216dfa2655 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } health_report { overall_health: HEALTHY } } }
I20260319 16:33:50.121042 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.119201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.24.187.193:39655)
0319 16:33:50.119255 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.119257 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.119265 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:50.119295 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.119296 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.119310 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.119355 (+ 45us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:50.119590 (+ 235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.120065 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.120589 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.120730 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.120760 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:50.120844 (+ 84us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:50.120903 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.120959 (+ 56us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.120981 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:33:50.122443 1475 raft_consensus.cc:1275] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Refusing update from remote peer 7cefd550566a43b7853389216dfa2655: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:50.122455 1332 raft_consensus.cc:1275] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Refusing update from remote peer 7cefd550566a43b7853389216dfa2655: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:50.122717 1561 consensus_queue.cc:1048] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:50.122853 1556 consensus_queue.cc:1048] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:50.124830 1564 mvcc.cc:204] Tried to move back new op lower bound from 7266050171379478528 to 7266050171353686016. Current Snapshot: MvccSnapshot[applied={T|T < 7266050171379478528}]
I20260319 16:33:50.124883 1566 mvcc.cc:204] Tried to move back new op lower bound from 7266050171379478528 to 7266050171353686016. Current Snapshot: MvccSnapshot[applied={T|T < 7266050171379478528}]
I20260319 16:33:50.124943 1565 mvcc.cc:204] Tried to move back new op lower bound from 7266050171379478528 to 7266050171353686016. Current Snapshot: MvccSnapshot[applied={T|T < 7266050171379478528}]
I20260319 16:33:50.127444 1030 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:50.127529 1030 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:50.128176 1030 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Sending DeleteTablet for 3 replicas of tablet 9608adfe31074b108e6c042dc4e429d2
I20260319 16:33:50.128463 1312 tablet_service.cc:1558] Processing DeleteTablet for tablet 9608adfe31074b108e6c042dc4e429d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:50 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:51066
I20260319 16:33:50.128535 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 9608adfe31074b108e6c042dc4e429d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:50 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:56220
I20260319 16:33:50.128701 1574 tablet_replica.cc:333] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: stopping tablet replica
I20260319 16:33:50.128717 1170 tablet_service.cc:1558] Processing DeleteTablet for tablet 9608adfe31074b108e6c042dc4e429d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:50 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:38352
I20260319 16:33:50.128705 1030 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260319 16:33:50.128813 1574 raft_consensus.cc:2243] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260319 16:33:50.128832 1030 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:50.128898 1574 raft_consensus.cc:2272] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:50.129277 1576 tablet_replica.cc:333] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: stopping tablet replica
I20260319 16:33:50.129344 1574 ts_tablet_manager.cc:1916] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:50.129408 1576 raft_consensus.cc:2243] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:50.129573 1576 raft_consensus.cc:2272] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:50.130069 1576 ts_tablet_manager.cc:1916] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:50.130504 1170 tablet_service.cc:1511] Processing CreateTablet for tablet fcfa4d72880841d992b2d0dd90c9ef3d (DEFAULT_TABLE table=test-table [id=37c1bcb365a446aa9f5c612316f7e688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:50.130607 1170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcfa4d72880841d992b2d0dd90c9ef3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.131002 1574 ts_tablet_manager.cc:1929] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:50.131078 1455 tablet_service.cc:1511] Processing CreateTablet for tablet fcfa4d72880841d992b2d0dd90c9ef3d (DEFAULT_TABLE table=test-table [id=37c1bcb365a446aa9f5c612316f7e688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:50.131145 1574 log.cc:1199] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/9608adfe31074b108e6c042dc4e429d2
I20260319 16:33:50.131181 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcfa4d72880841d992b2d0dd90c9ef3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.131402 1574 ts_tablet_manager.cc:1950] T 9608adfe31074b108e6c042dc4e429d2 P 68266dcfdb2e440da4de02e54e829c8d: Deleting consensus metadata
I20260319 16:33:50.131690 1576 ts_tablet_manager.cc:1929] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:50.131747 1576 log.cc:1199] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/9608adfe31074b108e6c042dc4e429d2
I20260319 16:33:50.131978 1576 ts_tablet_manager.cc:1950] T 9608adfe31074b108e6c042dc4e429d2 P 7cefd550566a43b7853389216dfa2655: Deleting consensus metadata
I20260319 16:33:50.132295 1016 catalog_manager.cc:5002] TS 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655): tablet 9608adfe31074b108e6c042dc4e429d2 (table test-table [id=fc2068564a7b4be09ca90cde275450df]) successfully deleted
I20260319 16:33:50.132457 1550 tablet_bootstrap.cc:492] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Bootstrap starting.
I20260319 16:33:50.132530 1016 catalog_manager.cc:5002] TS 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279): tablet 9608adfe31074b108e6c042dc4e429d2 (table test-table [id=fc2068564a7b4be09ca90cde275450df]) successfully deleted
I20260319 16:33:50.132717 1312 tablet_service.cc:1511] Processing CreateTablet for tablet fcfa4d72880841d992b2d0dd90c9ef3d (DEFAULT_TABLE table=test-table [id=37c1bcb365a446aa9f5c612316f7e688]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:50.132813 1312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcfa4d72880841d992b2d0dd90c9ef3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.132876 1550 tablet_bootstrap.cc:654] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.133623 1550 tablet_bootstrap.cc:492] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: No bootstrap required, opened a new log
I20260319 16:33:50.133685 1550 ts_tablet_manager.cc:1403] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:50.133890 1554 tablet_bootstrap.cc:492] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: Bootstrap starting.
I20260319 16:33:50.133874 1550 raft_consensus.cc:359] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.133936 1550 raft_consensus.cc:385] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.133955 1550 raft_consensus.cc:740] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccbfff35ffa4b63b510e428d594920f, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.133996 1550 consensus_queue.cc:260] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.134071 1550 ts_tablet_manager.cc:1434] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:50.134276 1554 tablet_bootstrap.cc:654] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.135191 1575 tablet_replica.cc:333] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: stopping tablet replica
I20260319 16:33:50.135257 1551 tablet_bootstrap.cc:492] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: Bootstrap starting.
I20260319 16:33:50.135314 1575 raft_consensus.cc:2243] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:50.135444 1575 raft_consensus.cc:2272] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:50.135687 1551 tablet_bootstrap.cc:654] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.135861 1575 ts_tablet_manager.cc:1916] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:50.136329 1554 tablet_bootstrap.cc:492] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: No bootstrap required, opened a new log
I20260319 16:33:50.136373 1554 ts_tablet_manager.cc:1403] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:33:50.136515 1554 raft_consensus.cc:359] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.136570 1554 raft_consensus.cc:385] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.136587 1554 raft_consensus.cc:740] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68266dcfdb2e440da4de02e54e829c8d, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.136627 1554 consensus_queue.cc:260] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.136713 1554 ts_tablet_manager.cc:1434] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:50.137207 1551 tablet_bootstrap.cc:492] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: No bootstrap required, opened a new log
I20260319 16:33:50.137251 1551 ts_tablet_manager.cc:1403] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260319 16:33:50.137377 1551 raft_consensus.cc:359] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.137429 1551 raft_consensus.cc:385] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.137447 1551 raft_consensus.cc:740] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.137485 1551 consensus_queue.cc:260] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.137562 1551 ts_tablet_manager.cc:1434] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:50.137600 1575 ts_tablet_manager.cc:1929] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:50.137650 1575 log.cc:1199] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/9608adfe31074b108e6c042dc4e429d2
I20260319 16:33:50.137895 1575 ts_tablet_manager.cc:1950] T 9608adfe31074b108e6c042dc4e429d2 P 6ccbfff35ffa4b63b510e428d594920f: Deleting consensus metadata
I20260319 16:33:50.138276 1016 catalog_manager.cc:5002] TS 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561): tablet 9608adfe31074b108e6c042dc4e429d2 (table test-table [id=fc2068564a7b4be09ca90cde275450df]) successfully deleted
W20260319 16:33:50.187100 1523 tablet.cc:2379] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:33:50.467558 1561 raft_consensus.cc:493] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:50.467666 1561 raft_consensus.cc:515] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.467836 1561 leader_election.cc:290] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:50.468078 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcfa4d72880841d992b2d0dd90c9ef3d" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d" is_pre_election: true
I20260319 16:33:50.468091 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcfa4d72880841d992b2d0dd90c9ef3d" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f" is_pre_election: true
I20260319 16:33:50.468158 1332 raft_consensus.cc:2468] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 0.
I20260319 16:33:50.468186 1475 raft_consensus.cc:2468] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 0.
I20260319 16:33:50.468323 1122 leader_election.cc:304] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68266dcfdb2e440da4de02e54e829c8d, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:50.468488 1561 raft_consensus.cc:2804] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:50.468539 1561 raft_consensus.cc:493] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:50.468554 1561 raft_consensus.cc:3060] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.469302 1561 raft_consensus.cc:515] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.469413 1561 leader_election.cc:290] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 election: Requested vote from peers 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:50.469604 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcfa4d72880841d992b2d0dd90c9ef3d" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d"
I20260319 16:33:50.469609 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcfa4d72880841d992b2d0dd90c9ef3d" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f"
I20260319 16:33:50.469666 1332 raft_consensus.cc:3060] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.469671 1475 raft_consensus.cc:3060] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.470261 1475 raft_consensus.cc:2468] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 1.
I20260319 16:33:50.470263 1332 raft_consensus.cc:2468] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 1.
I20260319 16:33:50.470444 1122 leader_election.cc:304] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68266dcfdb2e440da4de02e54e829c8d, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:50.470539 1561 raft_consensus.cc:2804] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:50.470603 1561 raft_consensus.cc:697] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 1 LEADER]: Becoming Leader. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Running, Role: LEADER
I20260319 16:33:50.470655 1561 consensus_queue.cc:237] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.471244 1030 catalog_manager.cc:5671] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cefd550566a43b7853389216dfa2655 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } health_report { overall_health: UNKNOWN } } }
Mar 19 16:33:50 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938029, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.254@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:50.521215 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.510460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46305 (local address 127.0.0.1:48258)
0319 16:33:50.510632 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:50.510646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:50.510737 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:50.510866 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:50.510866 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:50.511188 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:50.511353 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.511355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.512051 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.512052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:50.512427 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.512429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.512489 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.517463 (+ 4974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:50.517468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:50.518887 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:50.520348 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:50.520355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:50.520357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:50.520548 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:50.520682 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:50.520682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:50.520683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:50.520708 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:50.520945 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:50.520950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:50.520994 (+ 44us) client_negotiation.cc:769] Sending connection context
0319 16:33:50.521048 (+ 54us) client_negotiation.cc:240] Negotiation successful
0319 16:33:50.521141 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260319 16:33:50.521304 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.510564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.24.187.254:46305)
0319 16:33:50.510745 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.510747 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.510759 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:33:50.510784 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.510785 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.510796 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.510853 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:50.511443 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.511967 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.512510 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.512634 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.518977 (+ 6343us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:50.518981 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:50.518981 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:50.518989 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:50.520259 (+ 1270us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:50.520610 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:50.520611 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:50.520611 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:50.520627 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:50.520777 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:50.520779 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:50.520779 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:50.520884 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:50.520948 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.521072 (+ 124us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.521214 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":157,"thread_start_us":120,"threads_started":1}
I20260319 16:33:50.523248 1030 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:48258:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260319 16:33:50.523480 1030 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:50.525082 1170 tablet_service.cc:1511] Processing CreateTablet for tablet cc9d5e947acf406bac7443b292011485 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a96eb210a400403fb589a82023bb4cb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:33:50.525071 1312 tablet_service.cc:1511] Processing CreateTablet for tablet cc9d5e947acf406bac7443b292011485 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a96eb210a400403fb589a82023bb4cb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:33:50.525226 1312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9d5e947acf406bac7443b292011485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.525226 1170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9d5e947acf406bac7443b292011485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.525844 1455 tablet_service.cc:1511] Processing CreateTablet for tablet cc9d5e947acf406bac7443b292011485 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a96eb210a400403fb589a82023bb4cb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260319 16:33:50.525980 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9d5e947acf406bac7443b292011485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.526648 1551 tablet_bootstrap.cc:492] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: Bootstrap starting.
I20260319 16:33:50.526968 1551 tablet_bootstrap.cc:654] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.527194 1550 tablet_bootstrap.cc:492] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: Bootstrap starting.
I20260319 16:33:50.527225 1554 tablet_bootstrap.cc:492] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: Bootstrap starting.
I20260319 16:33:50.527581 1550 tablet_bootstrap.cc:654] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.527586 1554 tablet_bootstrap.cc:654] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.527647 1551 tablet_bootstrap.cc:492] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: No bootstrap required, opened a new log
I20260319 16:33:50.527688 1551 ts_tablet_manager.cc:1403] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:50.527801 1551 raft_consensus.cc:359] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.527861 1551 raft_consensus.cc:385] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.527878 1551 raft_consensus.cc:740] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.527928 1551 consensus_queue.cc:260] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.528015 1551 ts_tablet_manager.cc:1434] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:50.528036 1561 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.528119 1561 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:50.528390 1550 tablet_bootstrap.cc:492] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: No bootstrap required, opened a new log
I20260319 16:33:50.528426 1554 tablet_bootstrap.cc:492] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: No bootstrap required, opened a new log
I20260319 16:33:50.528438 1550 ts_tablet_manager.cc:1403] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:50.528460 1554 ts_tablet_manager.cc:1403] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:50.528595 1550 raft_consensus.cc:359] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.528651 1550 raft_consensus.cc:385] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.528622 1554 raft_consensus.cc:359] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.528669 1550 raft_consensus.cc:740] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccbfff35ffa4b63b510e428d594920f, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.528673 1554 raft_consensus.cc:385] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.528692 1554 raft_consensus.cc:740] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68266dcfdb2e440da4de02e54e829c8d, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.528712 1550 consensus_queue.cc:260] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.528728 1554 consensus_queue.cc:260] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.528784 1550 ts_tablet_manager.cc:1434] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:50.528810 1554 ts_tablet_manager.cc:1434] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:50.528882 1558 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.528924 1555 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.528966 1558 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:50.528987 1555 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:50.688607 1561 raft_consensus.cc:493] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:50.688714 1561 raft_consensus.cc:515] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.688858 1561 leader_election.cc:290] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:50.689203 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc9d5e947acf406bac7443b292011485" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d" is_pre_election: true
I20260319 16:33:50.689249 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc9d5e947acf406bac7443b292011485" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f" is_pre_election: true
I20260319 16:33:50.689339 1475 raft_consensus.cc:2468] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 0.
I20260319 16:33:50.689203 1558 raft_consensus.cc:493] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:50.689380 1558 raft_consensus.cc:515] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.689502 1122 leader_election.cc:304] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ccbfff35ffa4b63b510e428d594920f, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:50.689546 1332 raft_consensus.cc:2468] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 0.
I20260319 16:33:50.689617 1561 raft_consensus.cc:2804] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:50.689664 1561 raft_consensus.cc:493] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:50.689692 1561 raft_consensus.cc:3060] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.689764 1558 leader_election.cc:290] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:50.690634 1561 raft_consensus.cc:515] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.690784 1561 leader_election.cc:290] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 election: Requested vote from peers 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:50.690976 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc9d5e947acf406bac7443b292011485" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d"
I20260319 16:33:50.690989 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc9d5e947acf406bac7443b292011485" candidate_uuid: "7cefd550566a43b7853389216dfa2655" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f"
I20260319 16:33:50.691043 1332 raft_consensus.cc:3060] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.691056 1475 raft_consensus.cc:3060] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:50.691763 1475 raft_consensus.cc:2468] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 1.
I20260319 16:33:50.691812 1332 raft_consensus.cc:2468] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cefd550566a43b7853389216dfa2655 in term 1.
I20260319 16:33:50.691905 1122 leader_election.cc:304] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ccbfff35ffa4b63b510e428d594920f, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:50.691989 1561 raft_consensus.cc:2804] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:50.692040 1561 raft_consensus.cc:697] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [term 1 LEADER]: Becoming Leader. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Running, Role: LEADER
I20260319 16:33:50.692093 1561 consensus_queue.cc:237] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.692314 1556 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: TxnStatusTablet state changed. Reason: New leader 7cefd550566a43b7853389216dfa2655. Latest consensus state: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.692368 1556 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:33:50.692569 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.689934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:39655 (local address 127.24.187.194:55363)
0319 16:33:50.690034 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:50.690046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:50.690098 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:50.690252 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:50.690253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:50.690256 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:50.690505 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.690507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.691376 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.691377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:50.692348 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.692351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.692432 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.692437 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:50.692466 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:33:50.692506 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":61,"threads_started":1}
I20260319 16:33:50.692866 1030 catalog_manager.cc:5671] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cefd550566a43b7853389216dfa2655 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20260319 16:33:50.693130 1596 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260319 16:33:50.693248 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.690001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:55363 (local address 127.24.187.193:39655)
0319 16:33:50.690102 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.690104 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.690115 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:50.690164 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.690165 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.690175 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.690235 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:50.690577 (+ 342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.691290 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.692600 (+ 1310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.693075 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.693084 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.693106 (+ 22us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.693137 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
I20260319 16:33:50.693389 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.690205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:44935 (local address 127.24.187.195:42561)
0319 16:33:50.690314 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.690316 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.690452 (+ 136us) server_negotiation.cc:408] Connection header received
0319 16:33:50.690554 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.690555 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.690567 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.690635 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:50.690973 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.691679 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.692711 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.693151 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.693158 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.693292 (+ 134us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.693335 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260319 16:33:50.693485 1190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc9d5e947acf406bac7443b292011485" candidate_uuid: "68266dcfdb2e440da4de02e54e829c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cefd550566a43b7853389216dfa2655" is_pre_election: true
I20260319 16:33:50.693583 1593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.690089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42561 (local address 127.24.187.194:44935)
0319 16:33:50.690380 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:50.690389 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:50.690475 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:50.690638 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:50.690639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:50.690641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:50.690857 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.690858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.691790 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.691791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:50.692614 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.692618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.693201 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.693206 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:50.693481 (+ 275us) client_negotiation.cc:240] Negotiation successful
0319 16:33:50.693523 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":34,"threads_started":1}
I20260319 16:33:50.693287 1596 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260319 16:33:50.693930 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc9d5e947acf406bac7443b292011485" candidate_uuid: "68266dcfdb2e440da4de02e54e829c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f" is_pre_election: true
I20260319 16:33:50.694025 1475 raft_consensus.cc:2393] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68266dcfdb2e440da4de02e54e829c8d in current term 1: Already voted for candidate 7cefd550566a43b7853389216dfa2655 in this term.
I20260319 16:33:50.694187 1264 leader_election.cc:304] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68266dcfdb2e440da4de02e54e829c8d; no voters: 6ccbfff35ffa4b63b510e428d594920f, 7cefd550566a43b7853389216dfa2655
I20260319 16:33:50.694305 1558 raft_consensus.cc:2749] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260319 16:33:50.710269 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.708403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.24.187.193:39655)
0319 16:33:50.708461 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.708463 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.708472 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:50.708512 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.708513 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.708523 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.708565 (+ 42us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:50.708811 (+ 246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.709254 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.709863 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.710005 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.710022 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:50.710045 (+ 23us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:50.710089 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.710167 (+ 78us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.710190 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:33:50.710271 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.708326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:39655 (local address 127.0.0.1:38378)
0319 16:33:50.708395 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:50.708406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:50.708456 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:50.708576 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:50.708577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:50.708579 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:50.708745 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.708747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.709320 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.709321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:50.709788 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.709791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.709854 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.709875 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:50.710101 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:50.710111 (+ 10us) client_negotiation.cc:769] Sending connection context
0319 16:33:50.710154 (+ 43us) client_negotiation.cc:240] Negotiation successful
0319 16:33:50.710204 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260319 16:33:50.711344 1332 raft_consensus.cc:1275] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Refusing update from remote peer 7cefd550566a43b7853389216dfa2655: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:50.711349 1475 raft_consensus.cc:1275] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Refusing update from remote peer 7cefd550566a43b7853389216dfa2655: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:50.711586 1561 consensus_queue.cc:1048] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:50.711704 1556 consensus_queue.cc:1048] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:50.712303 1558 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: TxnStatusTablet state changed. Reason: New leader 7cefd550566a43b7853389216dfa2655. Latest consensus state: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.712455 1558 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:50.712423 1555 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: TxnStatusTablet state changed. Reason: New leader 7cefd550566a43b7853389216dfa2655. Latest consensus state: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.712496 1555 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:50.714519 1561 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.714609 1561 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:33:50.714897 1561 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.714965 1561 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 7cefd550566a43b7853389216dfa2655: This TxnStatusTablet replica's current role is: LEADER
I20260319 16:33:50.715356 1558 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.715436 1558 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 68266dcfdb2e440da4de02e54e829c8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:50.715559 1555 tablet_replica.cc:442] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cefd550566a43b7853389216dfa2655" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } } }
I20260319 16:33:50.715632 1555 tablet_replica.cc:445] T cc9d5e947acf406bac7443b292011485 P 6ccbfff35ffa4b63b510e428d594920f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260319 16:33:50.721527 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.719370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:38489 (local address 127.24.187.193:39655)
0319 16:33:50.719411 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.719413 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.719544 (+ 131us) server_negotiation.cc:408] Connection header received
0319 16:33:50.719609 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.719610 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.719618 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.719661 (+ 43us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:50.719965 (+ 304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.720411 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.721171 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.721287 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.721301 (+ 14us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:50.721322 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:50.721364 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.721444 (+ 80us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.721463 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:33:50.721604 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.719309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:39655 (local address 127.24.187.193:38489)
0319 16:33:50.719479 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:50.719490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:50.719553 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:50.719679 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:50.719680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:50.719682 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:50.719909 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.719911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.720471 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.720471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:50.721101 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:50.721103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.721175 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.721193 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:50.721378 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:50.721382 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:33:50.721432 (+ 50us) client_negotiation.cc:240] Negotiation successful
0319 16:33:50.721473 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260319 16:33:50.723635 1475 raft_consensus.cc:1275] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Refusing update from remote peer 7cefd550566a43b7853389216dfa2655: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:50.723657 1332 raft_consensus.cc:1275] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Refusing update from remote peer 7cefd550566a43b7853389216dfa2655: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:50.723841 1610 consensus_queue.cc:1048] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:50.724212 1561 consensus_queue.cc:1048] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:50.740733 1031 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:50.740849 1031 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:50.741472 1031 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Sending DeleteTablet for 3 replicas of tablet fcfa4d72880841d992b2d0dd90c9ef3d
I20260319 16:33:50.741734 1170 tablet_service.cc:1558] Processing DeleteTablet for tablet fcfa4d72880841d992b2d0dd90c9ef3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:50 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:38352
I20260319 16:33:50.741829 1312 tablet_service.cc:1558] Processing DeleteTablet for tablet fcfa4d72880841d992b2d0dd90c9ef3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:50 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:51066
I20260319 16:33:50.742091 1615 tablet_replica.cc:333] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: stopping tablet replica
I20260319 16:33:50.742151 1615 raft_consensus.cc:2243] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:50.742209 1615 raft_consensus.cc:2272] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:50.742316 1616 tablet_replica.cc:333] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: stopping tablet replica
I20260319 16:33:50.742406 1616 raft_consensus.cc:2243] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:50.742441 1616 raft_consensus.cc:2272] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:50.742707 1615 ts_tablet_manager.cc:1916] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:50.742720 1616 ts_tablet_manager.cc:1916] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:50.744438 1616 ts_tablet_manager.cc:1929] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:50.744534 1615 ts_tablet_manager.cc:1929] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:50.744578 1616 log.cc:1199] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/fcfa4d72880841d992b2d0dd90c9ef3d
I20260319 16:33:50.744706 1615 log.cc:1199] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/fcfa4d72880841d992b2d0dd90c9ef3d
I20260319 16:33:50.744709 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet fcfa4d72880841d992b2d0dd90c9ef3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:50 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:56220
I20260319 16:33:50.744895 1619 tablet_replica.cc:333] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: stopping tablet replica
I20260319 16:33:50.744951 1615 ts_tablet_manager.cc:1950] T fcfa4d72880841d992b2d0dd90c9ef3d P 7cefd550566a43b7853389216dfa2655: Deleting consensus metadata
I20260319 16:33:50.745002 1619 raft_consensus.cc:2243] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:50.745007 1616 ts_tablet_manager.cc:1950] T fcfa4d72880841d992b2d0dd90c9ef3d P 68266dcfdb2e440da4de02e54e829c8d: Deleting consensus metadata
I20260319 16:33:50.745079 1619 raft_consensus.cc:2272] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:50.745311 1016 catalog_manager.cc:5002] TS 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655): tablet fcfa4d72880841d992b2d0dd90c9ef3d (table test-table [id=37c1bcb365a446aa9f5c612316f7e688]) successfully deleted
I20260319 16:33:50.745349 1619 ts_tablet_manager.cc:1916] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:50.745391 1016 catalog_manager.cc:5002] TS 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279): tablet fcfa4d72880841d992b2d0dd90c9ef3d (table test-table [id=37c1bcb365a446aa9f5c612316f7e688]) successfully deleted
I20260319 16:33:50.746649 1619 ts_tablet_manager.cc:1929] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:50.746701 1619 log.cc:1199] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/fcfa4d72880841d992b2d0dd90c9ef3d
I20260319 16:33:50.746896 1619 ts_tablet_manager.cc:1950] T fcfa4d72880841d992b2d0dd90c9ef3d P 6ccbfff35ffa4b63b510e428d594920f: Deleting consensus metadata
I20260319 16:33:50.747165 1016 catalog_manager.cc:5002] TS 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561): tablet fcfa4d72880841d992b2d0dd90c9ef3d (table test-table [id=37c1bcb365a446aa9f5c612316f7e688]) successfully deleted
Mar 19 16:33:50 dist-test-slave-svn8 krb5kdc[983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Mar 19 16:33:50 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:50 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
W20260319 16:33:50.768064 1587 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:50.762645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.24.187.254:46305)
0319 16:33:50.762724 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.762727 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.762816 (+ 89us) server_negotiation.cc:408] Connection header received
0319 16:33:50.762893 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.762894 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.762903 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.762954 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:50.763822 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.764316 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.765123 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.765251 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.767995 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48264: BlockingRecv error: recv got EOF from 127.0.0.1:48264 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Mar 19 16:33:50 dist-test-slave-svn8 krb5kdc[983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM
I20260319 16:33:50.775367 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:50.768821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.24.187.254:46305)
0319 16:33:50.768865 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:50.768867 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:50.768879 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:33:50.768947 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:50.768948 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:50.768957 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:50.769009 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:50.769611 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.770134 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:50.770620 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:50.770772 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:50.772794 (+ 2022us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:50.772797 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:50.772797 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:50.772804 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:50.774381 (+ 1577us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:50.774724 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:50.774726 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:50.774726 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:50.774754 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:50.774891 (+ 137us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:50.774892 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:50.774892 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:50.774997 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:50.775091 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:50.775186 (+ 95us) server_negotiation.cc:300] Negotiation successful
0319 16:33:50.775279 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:33:50.776819 1031 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:50.776973 1031 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:50.778347 1312 tablet_service.cc:1511] Processing CreateTablet for tablet 52a30f25ee604914a77e2151fb524361 (DEFAULT_TABLE table=test-table [id=b774617c12e344acb1855d7ea7cd92a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:50.778376 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 52a30f25ee604914a77e2151fb524361 (DEFAULT_TABLE table=test-table [id=b774617c12e344acb1855d7ea7cd92a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:50.778479 1312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a30f25ee604914a77e2151fb524361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.778479 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a30f25ee604914a77e2151fb524361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.779780 1554 tablet_bootstrap.cc:492] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: Bootstrap starting.
I20260319 16:33:50.779795 1170 tablet_service.cc:1511] Processing CreateTablet for tablet 52a30f25ee604914a77e2151fb524361 (DEFAULT_TABLE table=test-table [id=b774617c12e344acb1855d7ea7cd92a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:50.779901 1170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a30f25ee604914a77e2151fb524361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:50.780200 1554 tablet_bootstrap.cc:654] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.780865 1554 tablet_bootstrap.cc:492] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: No bootstrap required, opened a new log
I20260319 16:33:50.780922 1554 ts_tablet_manager.cc:1403] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:50.781062 1554 raft_consensus.cc:359] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.781133 1554 raft_consensus.cc:385] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.781150 1554 raft_consensus.cc:740] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68266dcfdb2e440da4de02e54e829c8d, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.781176 1554 consensus_queue.cc:260] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.781176 1551 tablet_bootstrap.cc:492] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: Bootstrap starting.
I20260319 16:33:50.781204 1550 tablet_bootstrap.cc:492] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: Bootstrap starting.
I20260319 16:33:50.781250 1554 ts_tablet_manager.cc:1434] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:50.781469 1551 tablet_bootstrap.cc:654] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.781632 1550 tablet_bootstrap.cc:654] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:50.782641 1550 tablet_bootstrap.cc:492] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: No bootstrap required, opened a new log
I20260319 16:33:50.782692 1550 ts_tablet_manager.cc:1403] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260319 16:33:50.782802 1551 tablet_bootstrap.cc:492] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: No bootstrap required, opened a new log
I20260319 16:33:50.782863 1551 ts_tablet_manager.cc:1403] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260319 16:33:50.782832 1550 raft_consensus.cc:359] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.782899 1550 raft_consensus.cc:385] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.782923 1550 raft_consensus.cc:740] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccbfff35ffa4b63b510e428d594920f, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.782963 1550 consensus_queue.cc:260] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.782974 1551 raft_consensus.cc:359] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.783051 1550 ts_tablet_manager.cc:1434] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:50.783053 1551 raft_consensus.cc:385] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:50.783080 1551 raft_consensus.cc:740] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Initialized, Role: FOLLOWER
I20260319 16:33:50.783128 1551 consensus_queue.cc:260] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:50.783215 1551 ts_tablet_manager.cc:1434] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260319 16:33:51.015343 1558 raft_consensus.cc:493] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:51.015470 1558 raft_consensus.cc:515] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.015630 1558 leader_election.cc:290] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:51.015893 1190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52a30f25ee604914a77e2151fb524361" candidate_uuid: "68266dcfdb2e440da4de02e54e829c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cefd550566a43b7853389216dfa2655" is_pre_election: true
I20260319 16:33:51.015892 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52a30f25ee604914a77e2151fb524361" candidate_uuid: "68266dcfdb2e440da4de02e54e829c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f" is_pre_election: true
I20260319 16:33:51.015990 1190 raft_consensus.cc:2468] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68266dcfdb2e440da4de02e54e829c8d in term 0.
I20260319 16:33:51.015995 1475 raft_consensus.cc:2468] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68266dcfdb2e440da4de02e54e829c8d in term 0.
I20260319 16:33:51.016140 1264 leader_election.cc:304] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68266dcfdb2e440da4de02e54e829c8d, 6ccbfff35ffa4b63b510e428d594920f; no voters:
I20260319 16:33:51.016243 1558 raft_consensus.cc:2804] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:51.016300 1558 raft_consensus.cc:493] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:51.016328 1558 raft_consensus.cc:3060] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:51.017084 1558 raft_consensus.cc:515] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.017201 1558 leader_election.cc:290] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [CANDIDATE]: Term 1 election: Requested vote from peers 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655), 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561)
I20260319 16:33:51.017410 1475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52a30f25ee604914a77e2151fb524361" candidate_uuid: "68266dcfdb2e440da4de02e54e829c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccbfff35ffa4b63b510e428d594920f"
I20260319 16:33:51.017386 1190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52a30f25ee604914a77e2151fb524361" candidate_uuid: "68266dcfdb2e440da4de02e54e829c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cefd550566a43b7853389216dfa2655"
I20260319 16:33:51.017486 1475 raft_consensus.cc:3060] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:51.017486 1190 raft_consensus.cc:3060] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:51.018258 1475 raft_consensus.cc:2468] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68266dcfdb2e440da4de02e54e829c8d in term 1.
I20260319 16:33:51.018332 1190 raft_consensus.cc:2468] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68266dcfdb2e440da4de02e54e829c8d in term 1.
I20260319 16:33:51.018491 1264 leader_election.cc:304] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68266dcfdb2e440da4de02e54e829c8d, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:51.018579 1558 raft_consensus.cc:2804] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:51.018748 1558 raft_consensus.cc:697] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 1 LEADER]: Becoming Leader. State: Replica: 68266dcfdb2e440da4de02e54e829c8d, State: Running, Role: LEADER
I20260319 16:33:51.018816 1558 consensus_queue.cc:237] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.019518 1031 catalog_manager.cc:5671] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194). New cstate: current_term: 1 leader_uuid: "68266dcfdb2e440da4de02e54e829c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20260319 16:33:51.068739 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.066522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.24.187.194:43279)
0319 16:33:51.066648 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:51.066650 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:51.066658 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:51.066709 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:51.066709 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:51.066726 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:51.066789 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:51.066993 (+ 204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.067769 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.068284 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.068412 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.068444 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:51.068538 (+ 94us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:51.068591 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:51.068639 (+ 48us) server_negotiation.cc:300] Negotiation successful
0319 16:33:51.068658 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260319 16:33:51.069684 1190 raft_consensus.cc:1275] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Refusing update from remote peer 68266dcfdb2e440da4de02e54e829c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:51.069695 1475 raft_consensus.cc:1275] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Refusing update from remote peer 68266dcfdb2e440da4de02e54e829c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:51.069970 1634 consensus_queue.cc:1048] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:51.070115 1558 consensus_queue.cc:1048] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:51.073272 1029 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:51.073354 1029 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:51.073961 1029 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Sending DeleteTablet for 3 replicas of tablet 52a30f25ee604914a77e2151fb524361
I20260319 16:33:51.074265 1170 tablet_service.cc:1558] Processing DeleteTablet for tablet 52a30f25ee604914a77e2151fb524361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:51 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:38352
I20260319 16:33:51.074319 1312 tablet_service.cc:1558] Processing DeleteTablet for tablet 52a30f25ee604914a77e2151fb524361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:51 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:51066
I20260319 16:33:51.074357 1615 tablet_replica.cc:333] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: stopping tablet replica
I20260319 16:33:51.074414 1615 raft_consensus.cc:2243] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:51.074445 1615 raft_consensus.cc:2272] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:51.074425 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 52a30f25ee604914a77e2151fb524361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:51 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:56220
I20260319 16:33:51.074505 1616 tablet_replica.cc:333] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: stopping tablet replica
I20260319 16:33:51.074555 1616 raft_consensus.cc:2243] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:51.074615 1616 raft_consensus.cc:2272] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:51.074690 1619 tablet_replica.cc:333] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: stopping tablet replica
I20260319 16:33:51.074746 1619 raft_consensus.cc:2243] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:51.074788 1619 raft_consensus.cc:2272] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:51.074787 1615 ts_tablet_manager.cc:1916] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:51.075107 1619 ts_tablet_manager.cc:1916] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:51.075193 1616 ts_tablet_manager.cc:1916] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:51.076567 1615 ts_tablet_manager.cc:1929] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:51.076632 1615 log.cc:1199] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/52a30f25ee604914a77e2151fb524361
I20260319 16:33:51.076761 1616 ts_tablet_manager.cc:1929] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:51.076815 1615 ts_tablet_manager.cc:1950] T 52a30f25ee604914a77e2151fb524361 P 7cefd550566a43b7853389216dfa2655: Deleting consensus metadata
I20260319 16:33:51.076819 1616 log.cc:1199] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/52a30f25ee604914a77e2151fb524361
I20260319 16:33:51.077018 1616 ts_tablet_manager.cc:1950] T 52a30f25ee604914a77e2151fb524361 P 68266dcfdb2e440da4de02e54e829c8d: Deleting consensus metadata
I20260319 16:33:51.077208 1016 catalog_manager.cc:5002] TS 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655): tablet 52a30f25ee604914a77e2151fb524361 (table test-table [id=b774617c12e344acb1855d7ea7cd92a5]) successfully deleted
I20260319 16:33:51.077296 1029 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:33:51.077461 1029 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:33:51.077680 1619 ts_tablet_manager.cc:1929] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:33:51.077764 1619 log.cc:1199] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/52a30f25ee604914a77e2151fb524361
I20260319 16:33:51.077963 1619 ts_tablet_manager.cc:1950] T 52a30f25ee604914a77e2151fb524361 P 6ccbfff35ffa4b63b510e428d594920f: Deleting consensus metadata
I20260319 16:33:51.078362 1016 catalog_manager.cc:5002] TS 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561): tablet 52a30f25ee604914a77e2151fb524361 (table test-table [id=b774617c12e344acb1855d7ea7cd92a5]) successfully deleted
I20260319 16:33:51.078449 1016 catalog_manager.cc:5002] TS 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279): tablet 52a30f25ee604914a77e2151fb524361 (table test-table [id=b774617c12e344acb1855d7ea7cd92a5]) successfully deleted
I20260319 16:33:51.079219 1170 tablet_service.cc:1511] Processing CreateTablet for tablet 4018dd5733064cf59319ca0d37041f26 (DEFAULT_TABLE table=test-table [id=98b4bc7c9b734f36aa46993dbb962313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:51.079244 1455 tablet_service.cc:1511] Processing CreateTablet for tablet 4018dd5733064cf59319ca0d37041f26 (DEFAULT_TABLE table=test-table [id=98b4bc7c9b734f36aa46993dbb962313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:51.079263 1312 tablet_service.cc:1511] Processing CreateTablet for tablet 4018dd5733064cf59319ca0d37041f26 (DEFAULT_TABLE table=test-table [id=98b4bc7c9b734f36aa46993dbb962313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:33:51.079329 1170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4018dd5733064cf59319ca0d37041f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:51.079342 1455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4018dd5733064cf59319ca0d37041f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:51.079353 1312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4018dd5733064cf59319ca0d37041f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:51.080492 1551 tablet_bootstrap.cc:492] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655: Bootstrap starting.
I20260319 16:33:51.080492 1554 tablet_bootstrap.cc:492] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: Bootstrap starting.
I20260319 16:33:51.080518 1550 tablet_bootstrap.cc:492] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: Bootstrap starting.
I20260319 16:33:51.080799 1551 tablet_bootstrap.cc:654] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:51.080822 1550 tablet_bootstrap.cc:654] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:51.080896 1554 tablet_bootstrap.cc:654] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:51.081547 1554 tablet_bootstrap.cc:492] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: No bootstrap required, opened a new log
I20260319 16:33:51.081591 1554 ts_tablet_manager.cc:1403] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:51.081739 1551 tablet_bootstrap.cc:492] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655: No bootstrap required, opened a new log
I20260319 16:33:51.081759 1550 tablet_bootstrap.cc:492] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: No bootstrap required, opened a new log
I20260319 16:33:51.081782 1551 ts_tablet_manager.cc:1403] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260319 16:33:51.081799 1550 ts_tablet_manager.cc:1403] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260319 16:33:51.081881 1551 raft_consensus.cc:359] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.081869 1554 raft_consensus.cc:359] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.081933 1551 raft_consensus.cc:385] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:51.081933 1554 raft_consensus.cc:385] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:51.081951 1554 raft_consensus.cc:740] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68266dcfdb2e440da4de02e54e829c8d, State: Initialized, Role: FOLLOWER
I20260319 16:33:51.081951 1551 raft_consensus.cc:740] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cefd550566a43b7853389216dfa2655, State: Initialized, Role: FOLLOWER
I20260319 16:33:51.081936 1550 raft_consensus.cc:359] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.081992 1550 raft_consensus.cc:385] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:51.081991 1554 consensus_queue.cc:260] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.081990 1551 consensus_queue.cc:260] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.082013 1550 raft_consensus.cc:740] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccbfff35ffa4b63b510e428d594920f, State: Initialized, Role: FOLLOWER
I20260319 16:33:51.082057 1550 consensus_queue.cc:260] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.082072 1554 ts_tablet_manager.cc:1434] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:51.082077 1551 ts_tablet_manager.cc:1434] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:51.082129 1550 ts_tablet_manager.cc:1434] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:51.432119 1555 raft_consensus.cc:493] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:33:51.432226 1555 raft_consensus.cc:515] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.432580 1555 leader_election.cc:290] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655), 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279)
I20260319 16:33:51.434954 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.432749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:39655 (local address 127.24.187.195:43117)
0319 16:33:51.432897 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:51.432910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:51.432970 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:51.433137 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:51.433138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:51.433140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:51.433353 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.433354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.434108 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.434110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:51.434700 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.434702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.434765 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.434771 (+ 6us) client_negotiation.cc:769] Sending connection context
0319 16:33:51.434798 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:33:51.434835 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260319 16:33:51.434962 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.432931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43279 (local address 127.24.187.195:42653)
0319 16:33:51.433024 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:51.433033 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:51.433079 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:51.433238 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:51.433239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:51.433240 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:33:51.433433 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.433434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.434163 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.434164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:51.434709 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.434711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.434791 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.434796 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:33:51.434826 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:33:51.434860 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":42,"threads_started":1}
I20260319 16:33:51.435346 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.433012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:42653 (local address 127.24.187.194:43279)
0319 16:33:51.433062 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:51.433064 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:51.433074 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:51.433153 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:51.433154 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:51.433168 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:51.433222 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:51.433500 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.434058 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.434790 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.435220 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.435229 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:51.435255 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:33:51.435288 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:33:51.435365 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.432877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:43117 (local address 127.24.187.193:39655)
0319 16:33:51.432997 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:51.432999 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:51.433017 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:33:51.433043 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:51.433045 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:51.433057 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:51.433129 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:33:51.433427 (+ 298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.434025 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.434782 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.435235 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.435244 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:51.435271 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:33:51.435302 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260319 16:33:51.435523 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4018dd5733064cf59319ca0d37041f26" candidate_uuid: "6ccbfff35ffa4b63b510e428d594920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d" is_pre_election: true
I20260319 16:33:51.435537 1190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4018dd5733064cf59319ca0d37041f26" candidate_uuid: "6ccbfff35ffa4b63b510e428d594920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cefd550566a43b7853389216dfa2655" is_pre_election: true
I20260319 16:33:51.435614 1332 raft_consensus.cc:2468] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ccbfff35ffa4b63b510e428d594920f in term 0.
I20260319 16:33:51.435614 1190 raft_consensus.cc:2468] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ccbfff35ffa4b63b510e428d594920f in term 0.
I20260319 16:33:51.435774 1407 leader_election.cc:304] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ccbfff35ffa4b63b510e428d594920f, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:51.435897 1555 raft_consensus.cc:2804] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:33:51.435940 1555 raft_consensus.cc:493] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:33:51.435956 1555 raft_consensus.cc:3060] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:51.436553 1555 raft_consensus.cc:515] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.436681 1555 leader_election.cc:290] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [CANDIDATE]: Term 1 election: Requested vote from peers 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655), 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279)
I20260319 16:33:51.436851 1190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4018dd5733064cf59319ca0d37041f26" candidate_uuid: "6ccbfff35ffa4b63b510e428d594920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cefd550566a43b7853389216dfa2655"
I20260319 16:33:51.436879 1332 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4018dd5733064cf59319ca0d37041f26" candidate_uuid: "6ccbfff35ffa4b63b510e428d594920f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68266dcfdb2e440da4de02e54e829c8d"
I20260319 16:33:51.436926 1190 raft_consensus.cc:3060] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:51.436933 1332 raft_consensus.cc:3060] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:51.437624 1332 raft_consensus.cc:2468] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ccbfff35ffa4b63b510e428d594920f in term 1.
I20260319 16:33:51.437654 1190 raft_consensus.cc:2468] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ccbfff35ffa4b63b510e428d594920f in term 1.
I20260319 16:33:51.437813 1407 leader_election.cc:304] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ccbfff35ffa4b63b510e428d594920f, 7cefd550566a43b7853389216dfa2655; no voters:
I20260319 16:33:51.437942 1555 raft_consensus.cc:2804] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:51.438079 1555 raft_consensus.cc:697] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 1 LEADER]: Becoming Leader. State: Replica: 6ccbfff35ffa4b63b510e428d594920f, State: Running, Role: LEADER
I20260319 16:33:51.438148 1555 consensus_queue.cc:237] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } }
I20260319 16:33:51.438817 1031 catalog_manager.cc:5671] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195). New cstate: current_term: 1 leader_uuid: "6ccbfff35ffa4b63b510e428d594920f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ccbfff35ffa4b63b510e428d594920f" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42561 } health_report { overall_health: HEALTHY } } }
I20260319 16:33:51.527114 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.524774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.24.187.195:42561)
0319 16:33:51.524964 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:51.524966 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:51.524977 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:33:51.525013 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:51.525014 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:51.525033 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:51.525096 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:51.525389 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.525962 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.526549 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.526697 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.526732 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:51.526823 (+ 91us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:51.526889 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:51.527018 (+ 129us) server_negotiation.cc:300] Negotiation successful
0319 16:33:51.527043 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260319 16:33:51.531925 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.529763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:37099 (local address 127.24.187.193:39655)
0319 16:33:51.529817 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:51.529819 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:51.529867 (+ 48us) server_negotiation.cc:408] Connection header received
0319 16:33:51.529910 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:51.529911 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:51.529920 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:51.529974 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:51.530310 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.530776 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.531516 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.531643 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.531660 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:51.531698 (+ 38us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:51.531748 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:51.531847 (+ 99us) server_negotiation.cc:300] Negotiation successful
0319 16:33:51.531867 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:33:51.531952 1654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.529644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:39655 (local address 127.24.187.195:37099)
0319 16:33:51.529805 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:51.529815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:51.529869 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:51.530033 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:51.530034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:51.530036 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:51.530199 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.530200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.530871 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.530872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:51.531428 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.531431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.531493 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.531508 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:51.531788 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:51.531790 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:33:51.531839 (+ 49us) client_negotiation.cc:240] Negotiation successful
0319 16:33:51.531879 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260319 16:33:51.534256 1190 raft_consensus.cc:1275] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Refusing update from remote peer 6ccbfff35ffa4b63b510e428d594920f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:51.534256 1332 raft_consensus.cc:1275] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Refusing update from remote peer 6ccbfff35ffa4b63b510e428d594920f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:33:51.534507 1647 consensus_queue.cc:1048] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [LEADER]: Connected to new peer: Peer: permanent_uuid: "68266dcfdb2e440da4de02e54e829c8d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:51.534657 1555 consensus_queue.cc:1048] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cefd550566a43b7853389216dfa2655" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:33:51.541595 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.539588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:33267 (local address 127.24.187.195:42561)
0319 16:33:51.539680 (+ 92us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:51.539683 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:51.539731 (+ 48us) server_negotiation.cc:408] Connection header received
0319 16:33:51.539788 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:51.539789 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:51.539801 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:51.539848 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:33:51.540087 (+ 239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.540580 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.541165 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.541344 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.541369 (+ 25us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:33:51.541397 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:33:51.541452 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:51.541517 (+ 65us) server_negotiation.cc:300] Negotiation successful
0319 16:33:51.541543 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260319 16:33:51.541628 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.539529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42561 (local address 127.24.187.193:33267)
0319 16:33:51.539678 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:51.539688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:51.539743 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:51.539851 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:51.539851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:51.539854 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0319 16:33:51.540010 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.540011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.540687 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:51.540689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:51.541079 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:51.541080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:51.541156 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:51.541162 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0319 16:33:51.541462 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0319 16:33:51.541465 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:33:51.541506 (+ 41us) client_negotiation.cc:240] Negotiation successful
0319 16:33:51.541557 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260319 16:33:51.548924 1029 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:51.549050 1029 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:33:51.549975 1029 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 268cc07ed887430cacdbb244cdd1ad75: Sending DeleteTablet for 3 replicas of tablet 4018dd5733064cf59319ca0d37041f26
I20260319 16:33:51.550251 1170 tablet_service.cc:1558] Processing DeleteTablet for tablet 4018dd5733064cf59319ca0d37041f26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:51 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:38352
I20260319 16:33:51.550267 1312 tablet_service.cc:1558] Processing DeleteTablet for tablet 4018dd5733064cf59319ca0d37041f26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:51 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:51066
I20260319 16:33:51.550367 1615 tablet_replica.cc:333] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655: stopping tablet replica
I20260319 16:33:51.550367 1616 tablet_replica.cc:333] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: stopping tablet replica
I20260319 16:33:51.550410 1615 raft_consensus.cc:2243] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:51.550417 1616 raft_consensus.cc:2243] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:33:51.550450 1615 raft_consensus.cc:2272] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:51.550453 1616 raft_consensus.cc:2272] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:51.550899 1615 ts_tablet_manager.cc:1916] T 4018dd5733064cf59319ca0d37041f26 P 7cefd550566a43b7853389216dfa2655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:51.551002 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 1105
I20260319 16:33:51.556222 1455 tablet_service.cc:1558] Processing DeleteTablet for tablet 4018dd5733064cf59319ca0d37041f26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:33:51 UTC) from {username='oryx', principal='oryx/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:56220
I20260319 16:33:51.556473 1619 tablet_replica.cc:333] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: stopping tablet replica
I20260319 16:33:51.556627 1619 raft_consensus.cc:2243] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:33:51.556761 1619 raft_consensus.cc:2272] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:33:51.557101 1619 ts_tablet_manager.cc:1916] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:51.558806 1619 ts_tablet_manager.cc:1929] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:51.558897 1619 log.cc:1199] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/4018dd5733064cf59319ca0d37041f26
I20260319 16:33:51.559129 1619 ts_tablet_manager.cc:1950] T 4018dd5733064cf59319ca0d37041f26 P 6ccbfff35ffa4b63b510e428d594920f: Deleting consensus metadata
I20260319 16:33:51.559654 1016 catalog_manager.cc:5002] TS 6ccbfff35ffa4b63b510e428d594920f (127.24.187.195:42561): tablet 4018dd5733064cf59319ca0d37041f26 (table test-table [id=98b4bc7c9b734f36aa46993dbb962313]) successfully deleted
I20260319 16:33:51.560344 1616 ts_tablet_manager.cc:1916] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:33:51.561774 1616 ts_tablet_manager.cc:1929] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260319 16:33:51.561843 1616 log.cc:1199] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/4018dd5733064cf59319ca0d37041f26
I20260319 16:33:51.562065 1616 ts_tablet_manager.cc:1950] T 4018dd5733064cf59319ca0d37041f26 P 68266dcfdb2e440da4de02e54e829c8d: Deleting consensus metadata
W20260319 16:33:51.562376 1016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:39655 (error 108)
I20260319 16:33:51.562538 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 1247
I20260319 16:33:51.562608 1016 catalog_manager.cc:5002] TS 68266dcfdb2e440da4de02e54e829c8d (127.24.187.194:43279): tablet 4018dd5733064cf59319ca0d37041f26 (table test-table [id=98b4bc7c9b734f36aa46993dbb962313]) successfully deleted
I20260319 16:33:51.563730 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:51.563560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:39655 (local address 127.0.0.1:38380)
0319 16:33:51.563656 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:51.563696 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
W20260319 16:33:51.563812 1016 catalog_manager.cc:4729] TS 7cefd550566a43b7853389216dfa2655 (127.24.187.193:39655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4018dd5733064cf59319ca0d37041f26: Network error: Client connection negotiation failed: client connection to 127.24.187.193:39655: connect: Connection refused (error 111)
I20260319 16:33:51.569550 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 1389
I20260319 16:33:51.574373 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 999
2026-03-19T16:33:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4276 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:51 dist-test-slave-svn8 krb5kdc[1666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:51 dist-test-slave-svn8 krb5kdc[1666](info): set up 2 sockets
Mar 19 16:33:51 dist-test-slave-svn8 krb5kdc[1666](info): commencing operation
krb5kdc: starting...
W20260319 16:33:53.415145 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.821s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:53 dist-test-slave-svn8 krb5kdc[1666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:53.552742 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:40513
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:40513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:53.627242 1682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:53.627398 1682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:53.627426 1682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:53.628891 1682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:53.628937 1682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:53.628949 1682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:53.628960 1682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:53.628970 1682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:53.630702 1682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:40513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:40513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:53.630899 1682 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:53.631181 1682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:53.633653 1688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:53.633683 1687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:53.633841 1682 server_base.cc:1060] running on GCE node
W20260319 16:33:53.633651 1690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:53.634115 1682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:53.634341 1682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:53.635504 1682 hybrid_clock.cc:648] HybridClock initialized: now 1773938033635484 us; error 27 us; skew 500 ppm
W20260319 16:33:53.637090 1682 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-03-19T16:33:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2065 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:33:53 dist-test-slave-svn8 krb5kdc[1695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:33:53 dist-test-slave-svn8 krb5kdc[1695](info): set up 2 sockets
Mar 19 16:33:53 dist-test-slave-svn8 krb5kdc[1695](info): commencing operation
krb5kdc: starting...
W20260319 16:33:55.478740 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.818s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:33:55 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:33:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:55.612072 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37313
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:55.685880 1711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:55.686046 1711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:55.686074 1711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:55.687520 1711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:55.687561 1711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:55.687579 1711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:55.687594 1711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:55.687609 1711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:55.689218 1711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:37313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:55.689440 1711 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:55.689594 1711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:55.691916 1716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:55.691937 1717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:55.691916 1719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:55.692026 1711 server_base.cc:1060] running on GCE node
I20260319 16:33:55.692312 1711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:55.692543 1711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:55.693748 1711 hybrid_clock.cc:648] HybridClock initialized: now 1773938035693696 us; error 31 us; skew 500 ppm
Mar 19 16:33:55 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:55.696125 1711 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:55.696646 1711 webserver.cc:492] Webserver started at http://127.24.187.254:39743/ using document root <none> and password file <none>
I20260319 16:33:55.696810 1711 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:55.696844 1711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:55.696954 1711 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:55.697942 1711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "df1889a797b741508cc2605624f73952"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "6557b38073a97190d39ab5b4eed411fd"
server_key_iv: "3252046002248f62e27f4ea4c4b7e905"
server_key_version: "encryptionkey@0"
I20260319 16:33:55.698235 1711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "df1889a797b741508cc2605624f73952"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "6557b38073a97190d39ab5b4eed411fd"
server_key_iv: "3252046002248f62e27f4ea4c4b7e905"
server_key_version: "encryptionkey@0"
I20260319 16:33:55.699365 1711 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:55.700273 1726 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:55.700454 1711 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:55.700511 1711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "df1889a797b741508cc2605624f73952"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "6557b38073a97190d39ab5b4eed411fd"
server_key_iv: "3252046002248f62e27f4ea4c4b7e905"
server_key_version: "encryptionkey@0"
I20260319 16:33:55.700582 1711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:55.728138 1711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:55.734965 1711 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:55.735095 1711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:55.738973 1711 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:37313
I20260319 16:33:55.739005 1779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:37313 every 8 connection(s)
I20260319 16:33:55.739290 1711 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:55.739799 1780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:33:55.742101 1780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Bootstrap starting.
I20260319 16:33:55.742627 1780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Neither blocks nor log segments found. Creating new log.
I20260319 16:33:55.742811 1780 log.cc:826] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:55.743368 1780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: No bootstrap required, opened a new log
I20260319 16:33:55.744606 1780 raft_consensus.cc:359] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:55.744722 1780 raft_consensus.cc:385] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:33:55.744742 1780 raft_consensus.cc:740] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1889a797b741508cc2605624f73952, State: Initialized, Role: FOLLOWER
I20260319 16:33:55.744829 1780 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:55.744891 1780 raft_consensus.cc:399] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:55.744927 1780 raft_consensus.cc:493] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:55.744958 1780 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:33:55.745438 1780 raft_consensus.cc:515] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:55.745525 1780 leader_election.cc:304] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df1889a797b741508cc2605624f73952; no voters:
I20260319 16:33:55.745639 1780 leader_election.cc:290] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:33:55.745743 1783 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:33:55.745857 1783 raft_consensus.cc:697] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 LEADER]: Becoming Leader. State: Replica: df1889a797b741508cc2605624f73952, State: Running, Role: LEADER
I20260319 16:33:55.745877 1780 sys_catalog.cc:565] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:55.745930 1783 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:55.746352 1784 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df1889a797b741508cc2605624f73952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } } }
I20260319 16:33:55.746482 1784 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:55.746389 1785 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1889a797b741508cc2605624f73952. Latest consensus state: current_term: 1 leader_uuid: "df1889a797b741508cc2605624f73952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } } }
I20260319 16:33:55.746553 1785 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:55.746603 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 1711
I20260319 16:33:55.746718 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:33:55.746732 1788 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:55.746923 25327 external_mini_cluster.cc:1468] Setting key 4f7d99aa59835bbaf9b09f9ec4fe3bd7
I20260319 16:33:55.747197 1788 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:55.748863 1788 catalog_manager.cc:1357] Generated new cluster ID: 714a4d30223e420e9aeba21174be1b1e
I20260319 16:33:55.748914 1788 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:33:55 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:55.757653 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:55.747555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.24.187.254:37313)
0319 16:33:55.748375 (+ 820us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:55.748385 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:55.748406 (+ 21us) server_negotiation.cc:408] Connection header received
0319 16:33:55.748503 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:55.748510 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:55.748558 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:55.748696 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:55.749612 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:55.750878 (+ 1266us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:55.751605 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:55.751857 (+ 252us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:55.754160 (+ 2303us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:55.754167 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:55.754167 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:55.754189 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:55.756216 (+ 2027us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:55.756575 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:55.756577 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:55.756578 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:55.756627 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:55.756777 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:55.756777 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:55.756778 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:55.756979 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:55.757089 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:55.757231 (+ 142us) server_negotiation.cc:300] Negotiation successful
0319 16:33:55.757342 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":69,"threads_started":1}
I20260319 16:33:55.767081 1788 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:33:55.767593 1788 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:55.781728 1788 catalog_manager.cc:6044] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Generated new TSK 0
I20260319 16:33:55.781899 1788 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:55.836405 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37313
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:55.910887 1808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:55.911038 1808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:55.911059 1808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:55.912451 1808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:55.912488 1808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:55.912513 1808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:55.914077 1808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:55.914234 1808 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:55.914353 1808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:55.916642 1814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:55.916647 1816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:55.916642 1813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:55.916970 1808 server_base.cc:1060] running on GCE node
I20260319 16:33:55.917107 1808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:55.917261 1808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:55.918382 1808 hybrid_clock.cc:648] HybridClock initialized: now 1773938035918365 us; error 28 us; skew 500 ppm
Mar 19 16:33:55 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:55.920576 1808 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:33:55.921088 1808 webserver.cc:492] Webserver started at http://127.24.187.193:41889/ using document root <none> and password file <none>
I20260319 16:33:55.921240 1808 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:55.921270 1808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:55.921343 1808 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:55.922219 1808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "5bcfe92a40294cbdbf1f46250d4378d7"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "d6c4ca5bf074a5a5457cbb0f9333b3d8"
server_key_iv: "478ec273f58557b17316c26dbfbaad0f"
server_key_version: "encryptionkey@0"
I20260319 16:33:55.922483 1808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "5bcfe92a40294cbdbf1f46250d4378d7"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "d6c4ca5bf074a5a5457cbb0f9333b3d8"
server_key_iv: "478ec273f58557b17316c26dbfbaad0f"
server_key_version: "encryptionkey@0"
I20260319 16:33:55.923589 1808 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:55.924273 1823 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:55.924458 1808 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:55.924520 1808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "5bcfe92a40294cbdbf1f46250d4378d7"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "d6c4ca5bf074a5a5457cbb0f9333b3d8"
server_key_iv: "478ec273f58557b17316c26dbfbaad0f"
server_key_version: "encryptionkey@0"
I20260319 16:33:55.924580 1808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:55.937659 1808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:55.939378 1808 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:55.939508 1808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:55.939692 1808 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:55.939992 1808 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:55.940030 1808 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:55.940062 1808 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:55.940086 1808 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:55.945533 1808 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:33531
I20260319 16:33:55.945588 1938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:33531 every 8 connection(s)
I20260319 16:33:55.945861 1808 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:33:55 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938035, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:55.950863 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 1808
I20260319 16:33:55.950960 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:33:55.951112 25327 external_mini_cluster.cc:1468] Setting key fceee071da5e8f8f6f569125b91999f2
I20260319 16:33:55.953599 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:55.946515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:35747 (local address 127.24.187.254:37313)
0319 16:33:55.946566 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:55.946568 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:55.946699 (+ 131us) server_negotiation.cc:408] Connection header received
0319 16:33:55.947186 (+ 487us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:55.947188 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:55.947202 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:55.947259 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:55.947989 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:55.948483 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:55.948991 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:55.949117 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:55.951234 (+ 2117us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:55.951239 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:55.951239 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:55.951247 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:55.952499 (+ 1252us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:55.952840 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:55.952842 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:55.952842 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:55.952863 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:55.953015 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:55.953016 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:55.953016 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:55.953104 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:55.953159 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:55.953411 (+ 252us) server_negotiation.cc:300] Negotiation successful
0319 16:33:55.953505 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:33:55.953693 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:55.946452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:35747)
0319 16:33:55.946601 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:55.946627 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:55.947078 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:55.947310 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:55.947313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:55.947603 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:55.947913 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:55.947917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:55.948547 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:55.948547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:55.948914 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:55.948915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:55.948995 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:55.949790 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:55.949803 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:55.951126 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:55.952573 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:55.952579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:55.952581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:55.952778 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:55.952912 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:55.952913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:55.952913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:55.952967 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:55.953155 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:55.953159 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:55.953290 (+ 131us) client_negotiation.cc:769] Sending connection context
0319 16:33:55.953379 (+ 89us) client_negotiation.cc:240] Negotiation successful
0319 16:33:55.953508 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260319 16:33:55.954152 1939 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37313
I20260319 16:33:55.954231 1939 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:55.954399 1939 heartbeater.cc:507] Master 127.24.187.254:37313 requested a full tablet report, sending...
I20260319 16:33:55.954975 1743 ts_manager.cc:194] Registered new tserver with Master: 5bcfe92a40294cbdbf1f46250d4378d7 (127.24.187.193:33531)
I20260319 16:33:55.955924 1743 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:35747
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:55.997252 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37313
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:56.072862 1946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:56.073031 1946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:56.073055 1946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:56.074731 1946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:56.074782 1946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:56.074808 1946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:56.076443 1946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:56.076668 1946 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:56.076845 1946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:56.079317 1952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.079293 1951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:56.079370 1946 server_base.cc:1060] running on GCE node
W20260319 16:33:56.079304 1954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:56.079696 1946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:56.079866 1946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:56.081058 1946 hybrid_clock.cc:648] HybridClock initialized: now 1773938036081041 us; error 30 us; skew 500 ppm
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:56.083607 1946 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:33:56.084132 1946 webserver.cc:492] Webserver started at http://127.24.187.194:45851/ using document root <none> and password file <none>
I20260319 16:33:56.084295 1946 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:56.084329 1946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:56.084417 1946 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:56.085388 1946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "c3861ad529f948d1ad16a6473e5a0b0b"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "61d1ee26b303fc4117b973680e93721a"
server_key_iv: "09528b8c4186fc4c9be33306444426e3"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.085688 1946 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "c3861ad529f948d1ad16a6473e5a0b0b"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "61d1ee26b303fc4117b973680e93721a"
server_key_iv: "09528b8c4186fc4c9be33306444426e3"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.086874 1946 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.087585 1961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.087749 1946 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:56.087805 1946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "c3861ad529f948d1ad16a6473e5a0b0b"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "61d1ee26b303fc4117b973680e93721a"
server_key_iv: "09528b8c4186fc4c9be33306444426e3"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.087867 1946 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:56.102659 1946 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:56.104548 1946 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:56.104671 1946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:56.104871 1946 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:56.105178 1946 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:56.105208 1946 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.105229 1946 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:56.105252 1946 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.110584 1946 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:40107
I20260319 16:33:56.110641 2075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:40107 every 8 connection(s)
I20260319 16:33:56.110901 1946 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:33:56.112085 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 1946
I20260319 16:33:56.112164 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:33:56.112334 25327 external_mini_cluster.cc:1468] Setting key 4bfbc40c9929d66b3d93594224b95830
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:56.119733 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.111547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:48599 (local address 127.24.187.254:37313)
0319 16:33:56.111604 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.111607 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.111772 (+ 165us) server_negotiation.cc:408] Connection header received
0319 16:33:56.112289 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.112290 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.112303 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.112386 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.113445 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.114143 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.114825 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.115002 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.117297 (+ 2295us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:56.117302 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:56.117303 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:56.117310 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:56.118626 (+ 1316us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:56.118969 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:56.118970 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:56.118971 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:56.118993 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:56.119141 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:56.119142 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:56.119142 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:56.119236 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:56.119291 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:56.119526 (+ 235us) server_negotiation.cc:300] Negotiation successful
0319 16:33:56.119638 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:33:56.119805 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.111490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:48599)
0319 16:33:56.111671 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.111697 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.112167 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.112446 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.112451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.112987 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.113330 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.113335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.114252 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.114253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.114726 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.114729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.114839 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.115817 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.115838 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.117205 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:56.118702 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:56.118707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:56.118710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:56.118905 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:56.119040 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:56.119041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:56.119042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:56.119094 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:56.119285 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:56.119289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:56.119410 (+ 121us) client_negotiation.cc:769] Sending connection context
0319 16:33:56.119501 (+ 91us) client_negotiation.cc:240] Negotiation successful
0319 16:33:56.119629 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260319 16:33:56.120182 2076 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37313
I20260319 16:33:56.120265 2076 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:56.120456 2076 heartbeater.cc:507] Master 127.24.187.254:37313 requested a full tablet report, sending...
I20260319 16:33:56.120823 1743 ts_manager.cc:194] Registered new tserver with Master: c3861ad529f948d1ad16a6473e5a0b0b (127.24.187.194:40107)
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
I20260319 16:33:56.121237 1743 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:48599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:33:56.160362 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37313
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:56.234561 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:56.234736 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:56.234767 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:56.236286 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:56.236334 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:56.236375 2083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:56.238005 2083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:56.238224 2083 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:56.238394 2083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:56.240653 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.240665 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.240749 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:56.241285 2083 server_base.cc:1060] running on GCE node
I20260319 16:33:56.241469 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:56.241688 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:56.242895 2083 hybrid_clock.cc:648] HybridClock initialized: now 1773938036242867 us; error 28 us; skew 500 ppm
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:56.245191 2083 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:33:56.245703 2083 webserver.cc:492] Webserver started at http://127.24.187.195:36679/ using document root <none> and password file <none>
I20260319 16:33:56.245896 2083 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:56.245932 2083 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:56.246043 2083 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:33:56.247015 2083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "02aee01013904636ac663af7414c4017"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "0a79229620a2e9dd766d15f409a79974"
server_key_iv: "8c912ecd24e739a60aee4fcdf1c1b28f"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.247304 2083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "02aee01013904636ac663af7414c4017"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "0a79229620a2e9dd766d15f409a79974"
server_key_iv: "8c912ecd24e739a60aee4fcdf1c1b28f"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.248406 2083 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:33:56.249173 2098 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.249349 2083 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.249413 2083 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "02aee01013904636ac663af7414c4017"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "0a79229620a2e9dd766d15f409a79974"
server_key_iv: "8c912ecd24e739a60aee4fcdf1c1b28f"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.249482 2083 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:56.278556 2083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:56.280172 2083 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:56.280285 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:56.280490 2083 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:56.280829 2083 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:56.280862 2083 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.280885 2083 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:56.280908 2083 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.286124 2083 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:36147
I20260319 16:33:56.286201 2212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:36147 every 8 connection(s)
I20260319 16:33:56.286450 2083 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:33:56.294399 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.287105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:35551 (local address 127.24.187.254:37313)
0319 16:33:56.287154 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.287156 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.287311 (+ 155us) server_negotiation.cc:408] Connection header received
0319 16:33:56.287828 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.287830 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.287843 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.287893 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.288607 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.289079 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.289603 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.289799 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.292058 (+ 2259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:56.292062 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:56.292063 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:56.292069 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:56.293290 (+ 1221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:56.293641 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:56.293642 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:56.293643 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:56.293666 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:56.293849 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:56.293849 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:56.293850 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:56.293941 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:56.293998 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:56.294220 (+ 222us) server_negotiation.cc:300] Negotiation successful
0319 16:33:56.294306 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:33:56.294502 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.287044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:35551)
0319 16:33:56.287202 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.287235 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.287731 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.287955 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.287959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.288249 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.288511 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.288513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.289156 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.289157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.289524 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.289525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.289636 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.290562 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.290576 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.291963 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:33:56.293370 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:56.293375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:56.293378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:56.293573 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:56.293736 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:33:56.293737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:33:56.293737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:33:56.293797 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:33:56.293991 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:33:56.293996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:33:56.294107 (+ 111us) client_negotiation.cc:769] Sending connection context
0319 16:33:56.294197 (+ 90us) client_negotiation.cc:240] Negotiation successful
0319 16:33:56.294319 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260319 16:33:56.294912 2213 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37313
I20260319 16:33:56.295029 2213 heartbeater.cc:461] Registering TS with master...
I20260319 16:33:56.295115 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2083
I20260319 16:33:56.295207 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:33:56.295243 2213 heartbeater.cc:507] Master 127.24.187.254:37313 requested a full tablet report, sending...
I20260319 16:33:56.295401 25327 external_mini_cluster.cc:1468] Setting key 205308bc0a88c3f75c473fde238db35e
I20260319 16:33:56.295670 1743 ts_manager.cc:194] Registered new tserver with Master: 02aee01013904636ac663af7414c4017 (127.24.187.195:36147)
I20260319 16:33:56.296103 1743 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:35551
I20260319 16:33:56.296177 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.193.keytab.
Entry for principal oryx/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.193.keytab.
WARNING: no policy specified for oryx/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.194.keytab.
Entry for principal oryx/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.194.keytab.
WARNING: no policy specified for oryx/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.195.keytab.
Entry for principal oryx/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.195.keytab.
I20260319 16:33:56.362006 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 1808
I20260319 16:33:56.366417 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 1946
I20260319 16:33:56.370630 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2083
I20260319 16:33:56.374603 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 1711
I20260319 16:33:56.379137 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37313
--webserver_interface=127.24.187.254
--webserver_port=39743
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:33:56.454201 2222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:56.454367 2222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:56.454388 2222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:56.455844 2222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:33:56.455883 2222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:56.455895 2222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:56.455906 2222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:33:56.455916 2222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:33:56.457557 2222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:37313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=39743
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:56.457773 2222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:56.457913 2222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:56.460247 2228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.460264 2230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.460264 2227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:56.460546 2222 server_base.cc:1060] running on GCE node
I20260319 16:33:56.460790 2222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:56.461004 2222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:56.462154 2222 hybrid_clock.cc:648] HybridClock initialized: now 1773938036462136 us; error 29 us; skew 500 ppm
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:56.464576 2222 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:33:56.465057 2222 webserver.cc:492] Webserver started at http://127.24.187.254:39743/ using document root <none> and password file <none>
I20260319 16:33:56.465212 2222 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:56.465260 2222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:56.466472 2222 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.467020 2237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.467173 2222 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:33:56.467228 2222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "df1889a797b741508cc2605624f73952"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "6557b38073a97190d39ab5b4eed411fd"
server_key_iv: "3252046002248f62e27f4ea4c4b7e905"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.467509 2222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:56.481952 2222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:56.484036 2222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:56.484175 2222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:56.487953 2222 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:37313
I20260319 16:33:56.487984 2290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:37313 every 8 connection(s)
I20260319 16:33:56.488276 2222 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:33:56.489437 2291 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:33:56.490046 2291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Bootstrap starting.
I20260319 16:33:56.493238 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2222
I20260319 16:33:56.493518 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:33531
--local_ip_for_outbound_sockets=127.24.187.193
--tserver_master_addrs=127.24.187.254:37313
--webserver_port=41889
--webserver_interface=127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:33:56.498534 2291 log.cc:826] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Log is configured to *not* fsync() on all Append() calls
I20260319 16:33:56.500141 2291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:33:56.500422 2291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Bootstrap complete.
I20260319 16:33:56.502393 2291 raft_consensus.cc:359] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:56.502593 2291 raft_consensus.cc:740] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1889a797b741508cc2605624f73952, State: Initialized, Role: FOLLOWER
I20260319 16:33:56.502727 2291 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:56.502794 2291 raft_consensus.cc:399] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:33:56.502839 2291 raft_consensus.cc:493] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:33:56.502887 2291 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 1 FOLLOWER]: Advancing to term 2
I20260319 16:33:56.503777 2291 raft_consensus.cc:515] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:56.503872 2291 leader_election.cc:304] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df1889a797b741508cc2605624f73952; no voters:
I20260319 16:33:56.504122 2291 leader_election.cc:290] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260319 16:33:56.504163 2295 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 2 FOLLOWER]: Leader election won for term 2
I20260319 16:33:56.504264 2295 raft_consensus.cc:697] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [term 2 LEADER]: Becoming Leader. State: Replica: df1889a797b741508cc2605624f73952, State: Running, Role: LEADER
I20260319 16:33:56.504345 2295 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } }
I20260319 16:33:56.504375 2291 sys_catalog.cc:565] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:33:56.505201 2296 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df1889a797b741508cc2605624f73952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } } }
I20260319 16:33:56.505280 2296 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:56.505581 2297 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1889a797b741508cc2605624f73952. Latest consensus state: current_term: 2 leader_uuid: "df1889a797b741508cc2605624f73952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1889a797b741508cc2605624f73952" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37313 } } }
I20260319 16:33:56.505646 2297 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952 [sys.catalog]: This master's current role is: LEADER
I20260319 16:33:56.505820 2305 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:33:56.506284 2305 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:33:56.506417 2305 catalog_manager.cc:1269] Loaded cluster ID: 714a4d30223e420e9aeba21174be1b1e
I20260319 16:33:56.506451 2305 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:33:56.507678 2305 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:33:56.507839 2305 catalog_manager.cc:6055] T 00000000000000000000000000000000 P df1889a797b741508cc2605624f73952: Loaded TSK: 0
I20260319 16:33:56.508021 2305 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260319 16:33:56.571336 2293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:56.571523 2293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:56.571556 2293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:56.573247 2293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:56.573314 2293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:56.573356 2293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:33:56.575121 2293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.193.keytab
--principal=oryx/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:33531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=41889
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:56.575356 2293 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:56.575528 2293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:56.578145 2317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.578138 2320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.578138 2318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:56.578671 2293 server_base.cc:1060] running on GCE node
I20260319 16:33:56.578857 2293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:56.579061 2293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:56.580215 2293 hybrid_clock.cc:648] HybridClock initialized: now 1773938036580199 us; error 28 us; skew 500 ppm
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:56.582979 2293 init.cc:377] Logged in from keytab as oryx/127.24.187.193@KRBTEST.COM (short username oryx)
I20260319 16:33:56.583506 2293 webserver.cc:492] Webserver started at http://127.24.187.193:41889/ using document root <none> and password file <none>
I20260319 16:33:56.583684 2293 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:56.583729 2293 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:56.584986 2293 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.585686 2327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.585933 2293 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.586014 2293 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "5bcfe92a40294cbdbf1f46250d4378d7"
format_stamp: "Formatted at 2026-03-19 16:33:55 on dist-test-slave-svn8"
server_key: "d6c4ca5bf074a5a5457cbb0f9333b3d8"
server_key_iv: "478ec273f58557b17316c26dbfbaad0f"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.586395 2293 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:56.619240 2293 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:56.621824 2293 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:56.621966 2293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:56.622203 2293 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:56.622576 2293 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:56.622627 2293 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.622659 2293 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:56.622674 2293 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.629765 2293 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:33531
I20260319 16:33:56.629819 2441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:33531 every 8 connection(s)
I20260319 16:33:56.630146 2293 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.638427 2444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.630866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:56961)
0319 16:33:56.631031 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.631068 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.631685 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.632048 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.632052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.632468 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.632840 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.632846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.633995 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.633996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.634595 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.634598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.634741 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.635767 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.635789 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.638251 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
W20260319 16:33:56.638653 2442 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:37313 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.187.254:37313: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
W20260319 16:33:56.638945 2445 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.630984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:56961 (local address 127.24.187.254:37313)
0319 16:33:56.631146 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.631154 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.631175 (+ 21us) server_negotiation.cc:408] Connection header received
0319 16:33:56.631856 (+ 681us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.631861 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.631897 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.631991 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.632979 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.633906 (+ 927us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.634708 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.634894 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.638401 (+ 3507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:56961: BlockingRecv error: recv got EOF from 127.24.187.193:56961 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260319 16:33:56.638947 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2293
I20260319 16:33:56.639240 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:40107
--local_ip_for_outbound_sockets=127.24.187.194
--tserver_master_addrs=127.24.187.254:37313
--webserver_port=45851
--webserver_interface=127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.643849 2444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.638907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:54243)
0319 16:33:56.639089 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.639100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.639150 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.639323 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.639325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.639478 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.639670 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.639672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.640563 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.640564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.641109 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.641112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.641179 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.641707 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.641752 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.643779 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167}
W20260319 16:33:56.644209 2445 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.638979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:54243 (local address 127.24.187.254:37313)
0319 16:33:56.639059 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.639061 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.639139 (+ 78us) server_negotiation.cc:408] Connection header received
0319 16:33:56.639200 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.639201 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.639212 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.639270 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.639760 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.640464 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.641345 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.641598 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.644152 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:54243: BlockingRecv error: recv got EOF from 127.24.187.193:54243 (error 108)
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":96,"threads_started":1}
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.649545 2444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.644262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:35743)
0319 16:33:56.644446 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.644456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.644511 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.644675 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.644676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.644818 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.644997 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.644999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.646025 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.646027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.646570 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.646574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.646912 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.647482 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.647487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.649475 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170}
W20260319 16:33:56.649662 2446 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.644368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:35743 (local address 127.24.187.254:37313)
0319 16:33:56.644415 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.644417 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.644513 (+ 96us) server_negotiation.cc:408] Connection header received
0319 16:33:56.644578 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.644579 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.644590 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.644660 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.645090 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.645929 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.646674 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.646859 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.649601 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:35743: BlockingRecv error: recv got EOF from 127.24.187.193:35743 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260319 16:33:56.649683 2442 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260319 16:33:56.717895 2447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:56.718075 2447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:56.718098 2447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:56.719530 2447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:56.719571 2447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:56.719595 2447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:33:56.721153 2447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.194.keytab
--principal=oryx/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:40107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=45851
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:56.721335 2447 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:56.721484 2447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:56.723945 2452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.723958 2455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.723960 2453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:56.724215 2447 server_base.cc:1060] running on GCE node
I20260319 16:33:56.724404 2447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:56.724606 2447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:56.725764 2447 hybrid_clock.cc:648] HybridClock initialized: now 1773938036725745 us; error 34 us; skew 500 ppm
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:56.728148 2447 init.cc:377] Logged in from keytab as oryx/127.24.187.194@KRBTEST.COM (short username oryx)
I20260319 16:33:56.728637 2447 webserver.cc:492] Webserver started at http://127.24.187.194:45851/ using document root <none> and password file <none>
I20260319 16:33:56.728783 2447 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:56.728829 2447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:56.730118 2447 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.730635 2462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.730811 2447 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.730868 2447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "c3861ad529f948d1ad16a6473e5a0b0b"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "61d1ee26b303fc4117b973680e93721a"
server_key_iv: "09528b8c4186fc4c9be33306444426e3"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.731127 2447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:56.747835 2447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:56.749540 2447 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:56.749646 2447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:56.749862 2447 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:56.750141 2447 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:56.750169 2447 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.750190 2447 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:56.750200 2447 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.755797 2447 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:40107
I20260319 16:33:56.755861 2575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:40107 every 8 connection(s)
I20260319 16:33:56.756125 2447 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.762689 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.756729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:58293)
0319 16:33:56.756927 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.756962 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.757490 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.757744 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.757748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.758046 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.758315 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.758318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.758976 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.758976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.759372 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.759373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.759490 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.760377 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.760393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.762572 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
W20260319 16:33:56.762749 2446 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.756812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:58293 (local address 127.24.187.254:37313)
0319 16:33:56.756889 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.756892 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.757054 (+ 162us) server_negotiation.cc:408] Connection header received
0319 16:33:56.757586 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.757587 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.757602 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.757657 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.758410 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.758897 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.759450 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.759619 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.762692 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:58293: BlockingRecv error: recv got EOF from 127.24.187.194:58293 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260319 16:33:56.762928 2576 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:37313 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.187.254:37313: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
I20260319 16:33:56.764143 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2447
I20260319 16:33:56.764448 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:36147
--local_ip_for_outbound_sockets=127.24.187.195
--tserver_master_addrs=127.24.187.254:37313
--webserver_port=36679
--webserver_interface=127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.767238 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.763135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:35999)
0319 16:33:56.763218 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.763230 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.763278 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.763454 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.763455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.763565 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.763700 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.763702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.764738 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.764739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.765094 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.765096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.765154 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.765550 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.765554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.767179 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260319 16:33:56.767339 2446 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.763224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:35999 (local address 127.24.187.254:37313)
0319 16:33:56.763273 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.763275 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.763284 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:56.763321 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.763322 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.763334 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.763379 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.763793 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.764493 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.765169 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.765331 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.767281 (+ 1950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:35999: BlockingRecv error: recv got EOF from 127.24.187.194:35999 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.771425 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.767603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:39531)
0319 16:33:56.767710 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.767721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.767776 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.767903 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.767904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.768034 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.768193 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.768195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.768781 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.768782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.769292 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.769294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.769364 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.769789 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.769794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.771366 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260319 16:33:56.771517 2446 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.767664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:39531 (local address 127.24.187.254:37313)
0319 16:33:56.767712 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.767713 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.767766 (+ 53us) server_negotiation.cc:408] Connection header received
0319 16:33:56.767833 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.767834 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.767843 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.767885 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.768253 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.768715 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.769380 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.769543 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.771466 (+ 1923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:39531: BlockingRecv error: recv got EOF from 127.24.187.194:39531 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260319 16:33:56.771632 2576 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260319 16:33:56.844332 2579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:33:56.844514 2579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:33:56.844548 2579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:33:56.846072 2579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:33:56.846118 2579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:33:56.846155 2579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:33:56.847821 2579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/krb5kdc/oryx_127.24.187.195.keytab
--principal=oryx/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:36147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=36679
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:33:56.848042 2579 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:33:56.848198 2579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:33:56.850746 2584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.850845 2586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:33:56.850764 2588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:33:56.851018 2579 server_base.cc:1060] running on GCE node
I20260319 16:33:56.851222 2579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:33:56.851416 2579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:33:56.852563 2579 hybrid_clock.cc:648] HybridClock initialized: now 1773938036852547 us; error 31 us; skew 500 ppm
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938036, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:33:56.855067 2579 init.cc:377] Logged in from keytab as oryx/127.24.187.195@KRBTEST.COM (short username oryx)
I20260319 16:33:56.855604 2579 webserver.cc:492] Webserver started at http://127.24.187.195:36679/ using document root <none> and password file <none>
I20260319 16:33:56.855767 2579 fs_manager.cc:362] Metadata directory not provided
I20260319 16:33:56.855816 2579 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:33:56.857033 2579 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.857606 2595 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.857792 2579 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:33:56.857861 2579 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "02aee01013904636ac663af7414c4017"
format_stamp: "Formatted at 2026-03-19 16:33:56 on dist-test-slave-svn8"
server_key: "0a79229620a2e9dd766d15f409a79974"
server_key_iv: "8c912ecd24e739a60aee4fcdf1c1b28f"
server_key_version: "encryptionkey@0"
I20260319 16:33:56.858132 2579 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:33:56.903268 2579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:33:56.905468 2579 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:33:56.905596 2579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:33:56.905897 2579 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:33:56.906307 2579 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:33:56.906347 2579 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.906370 2579 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:33:56.906380 2579 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:33:56.911587 2579 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:36147
I20260319 16:33:56.911657 2708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:36147 every 8 connection(s)
I20260319 16:33:56.911937 2579 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.918164 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.912577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:42485)
0319 16:33:56.912711 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.912742 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.913205 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.913449 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.913453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.913766 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.914040 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.914043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.914698 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.914699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.915059 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.915060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.915141 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.915960 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.915974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.918050 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
W20260319 16:33:56.918184 2446 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.912636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:42485 (local address 127.24.187.254:37313)
0319 16:33:56.912689 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.912690 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.912831 (+ 141us) server_negotiation.cc:408] Connection header received
0319 16:33:56.913300 (+ 469us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.913301 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.913313 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.913375 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.914127 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.914633 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.915147 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.915278 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.918139 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:42485: BlockingRecv error: recv got EOF from 127.24.187.195:42485 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260319 16:33:56.918380 2709 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:37313 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.187.254:37313: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
I20260319 16:33:56.919880 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2579
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.922508 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.918606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:54645)
0319 16:33:56.918648 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.918659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.918697 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.918924 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.918925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.919045 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.919179 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.919181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.919861 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.919863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.920301 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.920303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.920418 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.920811 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.920816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.922451 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260319 16:33:56.922595 2446 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.918761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:54645 (local address 127.24.187.254:37313)
0319 16:33:56.918804 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.918805 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.918813 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:33:56.918833 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.918834 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.918841 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.918886 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.919268 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.919771 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.920428 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.920580 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.922541 (+ 1961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:54645: BlockingRecv error: recv got EOF from 127.24.187.195:54645 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:56 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:56.925984 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.922808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:60943)
0319 16:33:56.922855 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:56.922862 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:56.922892 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:56.923021 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:56.923022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:56.923127 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:56.923231 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.923232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.923789 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.923790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:56.924188 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:56.924190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.924254 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.924543 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:56.924547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:56.925939 (+ 1392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260319 16:33:56.926065 2446 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:56.922880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:60943 (local address 127.24.187.254:37313)
0319 16:33:56.922936 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.922938 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.922944 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:33:56.922962 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.922963 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.922970 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.923012 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.923293 (+ 281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.923728 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.924275 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.924448 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.926013 (+ 1565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:60943: BlockingRecv error: recv got EOF from 127.24.187.195:60943 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260319 16:33:56.926206 2709 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260319 16:33:56.926638 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:56.920295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.24.187.254:37313)
0319 16:33:56.920352 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:56.920355 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:56.920500 (+ 145us) server_negotiation.cc:408] Connection header received
0319 16:33:56.920598 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:56.920599 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:56.920609 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:56.920656 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:56.921454 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.921979 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:56.922596 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:56.922755 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:56.923783 (+ 1028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:33:56.923789 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:33:56.923789 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:33:56.923814 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:33:56.925377 (+ 1563us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:56.925663 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:56.925666 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:56.925666 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:56.925754 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:33:56.925883 (+ 129us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:33:56.925884 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:33:56.925885 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:33:56.926097 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:33:56.926210 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0319 16:33:56.926445 (+ 235us) server_negotiation.cc:300] Negotiation successful
0319 16:33:56.926556 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Mar 19 16:33:57 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:57 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:57.654310 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:57.650067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:60465)
0319 16:33:57.650205 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:57.650217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:57.650274 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:57.650399 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:57.650400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:57.650546 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:57.650714 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:57.650716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.651371 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:57.651372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:57.651885 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:57.651888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.651949 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:57.652461 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:57.652466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:57.654253 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
W20260319 16:33:57.654448 2714 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:57.650161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:60465 (local address 127.24.187.254:37313)
0319 16:33:57.650262 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:57.650263 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:57.650273 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:33:57.650321 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:57.650322 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:57.650331 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:57.650384 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:57.650781 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.651289 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:57.651943 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.652103 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:57.654382 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:60465: BlockingRecv error: recv got EOF from 127.24.187.193:60465 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
Mar 19 16:33:57 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:57 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:57.776196 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:57.772020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:46841)
0319 16:33:57.772157 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:57.772169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:57.772229 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:57.772409 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:57.772411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:57.772567 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:57.772731 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:57.772733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.773469 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:57.773470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:57.773895 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:57.773897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.773961 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:57.774424 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:57.774428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:57.776122 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
W20260319 16:33:57.776275 2714 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:57.772067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:46841 (local address 127.24.187.254:37313)
0319 16:33:57.772118 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:57.772120 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:57.772250 (+ 130us) server_negotiation.cc:408] Connection header received
0319 16:33:57.772291 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:57.772292 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:57.772304 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:57.772361 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:57.772829 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.773337 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:57.773985 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.774151 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:57.776224 (+ 2073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:46841: BlockingRecv error: recv got EOF from 127.24.187.194:46841 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Mar 19 16:33:57 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:57 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:57.930513 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:57.926575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:57303)
0319 16:33:57.926721 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:57.926732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:57.926790 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:57.926899 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:57.926899 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:57.927048 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:57.927201 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:57.927203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.927817 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:57.927818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:57.928180 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:57.928181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.928233 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:57.928706 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:57.928710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:57.930441 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
W20260319 16:33:57.930598 2714 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:57.926647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:57303 (local address 127.24.187.254:37313)
0319 16:33:57.926708 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:57.926710 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:57.926784 (+ 74us) server_negotiation.cc:408] Connection header received
0319 16:33:57.926842 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:57.926843 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:57.926851 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:57.926889 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:57.927264 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.927756 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:57.928268 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:57.928455 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:57.930547 (+ 2092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:57303: BlockingRecv error: recv got EOF from 127.24.187.195:57303 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Mar 19 16:33:58 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:58 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:58.659309 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:58.654847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:41641)
0319 16:33:58.654970 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:58.654980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:58.655036 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:58.655171 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:58.655172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:58.655413 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:58.655601 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:58.655604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.656461 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:58.656462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:58.656857 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:58.656859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.656921 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:58.657483 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:58.657488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:58.659252 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
W20260319 16:33:58.659421 2718 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:58.654957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:41641 (local address 127.24.187.254:37313)
0319 16:33:58.655057 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:58.655059 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:58.655068 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:58.655093 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:58.655094 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:58.655103 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:58.655160 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:58.655706 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.656368 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:58.656946 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.657099 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:58.659354 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:41641: BlockingRecv error: recv got EOF from 127.24.187.193:41641 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
Mar 19 16:33:58 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:58 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:58.780583 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:58.776692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:45073)
0319 16:33:58.776813 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:58.776825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:58.776898 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:58.777020 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:58.777021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:58.777196 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:58.777345 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:58.777347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.778039 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:58.778040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:58.778404 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:58.778406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.778465 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:58.778900 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:58.778905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:58.780501 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
W20260319 16:33:58.780653 2718 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:58.776728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:45073 (local address 127.24.187.254:37313)
0319 16:33:58.776782 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:58.776784 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:58.776888 (+ 104us) server_negotiation.cc:408] Connection header received
0319 16:33:58.776949 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:58.776950 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:58.776965 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:58.777013 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:58.777427 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.777964 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:58.778477 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.778635 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:58.780597 (+ 1962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:45073: BlockingRecv error: recv got EOF from 127.24.187.194:45073 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Mar 19 16:33:58 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:58 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:58.935004 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:58.931009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:48273)
0319 16:33:58.931139 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:58.931149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:58.931209 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:58.931394 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:58.931395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:58.931556 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:58.931710 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:58.931711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.932352 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:58.932353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:58.932711 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:58.932712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.932770 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:58.933222 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:58.933227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:58.934918 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
W20260319 16:33:58.935093 2718 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:58.931131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:48273 (local address 127.24.187.254:37313)
0319 16:33:58.931196 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:58.931198 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:58.931207 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:58.931285 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:58.931286 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:58.931296 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:58.931339 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:58.931792 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.932271 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:58.932783 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:58.932944 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:58.935040 (+ 2096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:48273: BlockingRecv error: recv got EOF from 127.24.187.195:48273 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Mar 19 16:33:59 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:59 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:59.664168 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:59.659840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:47645)
0319 16:33:59.659963 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:59.659972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:59.660024 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:59.660196 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:59.660197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:59.660351 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:59.660537 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:59.660539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.661373 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:59.661374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:59.661809 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:59.661813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.661871 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:59.662340 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:59.662344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:59.664113 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
W20260319 16:33:59.664295 2722 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:59.659938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:47645 (local address 127.24.187.254:37313)
0319 16:33:59.660075 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:59.660077 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:59.660086 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:33:59.660109 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:59.660110 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:59.660120 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:59.660181 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:59.660603 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.661308 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:59.661935 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.662098 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:59.664231 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:47645: BlockingRecv error: recv got EOF from 127.24.187.193:47645 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
Mar 19 16:33:59 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:59 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:59.785174 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:59.781058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:60717)
0319 16:33:59.781196 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:59.781207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:59.781278 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:59.781482 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:59.781483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:59.781634 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:59.781870 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:59.781872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.782541 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:59.782543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:59.782951 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:59.782953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.783018 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:59.783553 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:59.783557 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:59.785116 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
W20260319 16:33:59.785292 2722 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:59.781138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:60717 (local address 127.24.187.254:37313)
0319 16:33:59.781189 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:59.781191 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:59.781319 (+ 128us) server_negotiation.cc:408] Connection header received
0319 16:33:59.781348 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:59.781349 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:59.781364 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:59.781472 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:59.781955 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.782449 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:59.783047 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.783183 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:59.785227 (+ 2044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:60717: BlockingRecv error: recv got EOF from 127.24.187.194:60717 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Mar 19 16:33:59 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:33:59 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:33:59.939471 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:33:59.935476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:45971)
0319 16:33:59.935611 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0319 16:33:59.935622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:33:59.935697 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:33:59.935807 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:33:59.935808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:33:59.935983 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:33:59.936140 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:59.936141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.936765 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:33:59.936765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:33:59.937133 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:33:59.937135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.937191 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:59.937668 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:33:59.937679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:33:59.939383 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
W20260319 16:33:59.939531 2722 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:33:59.935525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:45971 (local address 127.24.187.254:37313)
0319 16:33:59.935573 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:33:59.935575 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:33:59.935667 (+ 92us) server_negotiation.cc:408] Connection header received
0319 16:33:59.935740 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:33:59.935740 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:33:59.935748 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:33:59.935794 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:33:59.936200 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.936697 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:33:59.937211 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:33:59.937342 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:33:59.939481 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:45971: BlockingRecv error: recv got EOF from 127.24.187.195:45971 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Mar 19 16:34:00 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:00 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:00.668650 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:00.664665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:60265)
0319 16:34:00.664829 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:00.664859 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:00.664925 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:00.665067 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:00.665068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:00.665218 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:00.665417 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:00.665419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.666080 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:00.666081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:00.666491 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:00.666494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.666556 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:00.667004 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:00.667008 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:00.668591 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
W20260319 16:34:00.668794 2726 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:00.664819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:60265 (local address 127.24.187.254:37313)
0319 16:34:00.664958 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:00.664960 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:00.664968 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:34:00.664990 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:00.664991 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:00.665000 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:00.665058 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:00.665474 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.665991 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:00.666570 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.666702 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:00.668724 (+ 2022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:60265: BlockingRecv error: recv got EOF from 127.24.187.193:60265 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
Mar 19 16:34:00 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:00 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:00.792918 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:00.785665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:37097)
0319 16:34:00.785823 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:00.785834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:00.785890 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:00.786030 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:00.786031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:00.786186 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:00.786360 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:00.786362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.787094 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:00.787095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:00.787485 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:00.787486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.787548 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:00.791149 (+ 3601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:00.791155 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:00.792857 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
W20260319 16:34:00.793005 2726 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:00.785737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:37097 (local address 127.24.187.254:37313)
0319 16:34:00.785805 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:00.785807 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:00.785886 (+ 79us) server_negotiation.cc:408] Connection header received
0319 16:34:00.785946 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:00.785947 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:00.785958 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:00.786011 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:00.786473 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.787003 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:00.787577 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.787748 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:00.792949 (+ 5201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:37097: BlockingRecv error: recv got EOF from 127.24.187.194:37097 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Mar 19 16:34:00 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:00 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:00.944013 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:00.939965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:56523)
0319 16:34:00.940139 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:00.940151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:00.940217 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:00.940343 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:00.940344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:00.940504 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:00.940705 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:00.940706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.941315 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:00.941316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:00.941738 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:00.941740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.941796 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:00.942296 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:00.942301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:00.943955 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
W20260319 16:34:00.944151 2726 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:00.940097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:56523 (local address 127.24.187.254:37313)
0319 16:34:00.940164 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:00.940166 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:00.940199 (+ 33us) server_negotiation.cc:408] Connection header received
0319 16:34:00.940275 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:00.940275 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:00.940284 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:00.940333 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:00.940768 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.941242 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:00.941808 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:00.941931 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:00.944092 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:56523: BlockingRecv error: recv got EOF from 127.24.187.195:56523 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Mar 19 16:34:01 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:01 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:01.674059 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:01.669195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:43445)
0319 16:34:01.669403 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:01.669416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:01.669479 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:01.669737 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:01.669741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:01.669981 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:01.670195 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:01.670198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.671146 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:01.671148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:01.671600 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:01.671602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.671693 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:01.672221 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:01.672226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:01.674002 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
W20260319 16:34:01.674201 2730 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:01.669333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:43445 (local address 127.24.187.254:37313)
0319 16:34:01.669457 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:01.669459 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:01.669473 (+ 14us) server_negotiation.cc:408] Connection header received
0319 16:34:01.669543 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:01.669544 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:01.669553 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:01.669610 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:01.670335 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.671025 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:01.671703 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.671886 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:01.674129 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:43445: BlockingRecv error: recv got EOF from 127.24.187.193:43445 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
Mar 19 16:34:01 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:01 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:01.797830 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:01.793512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:38367)
0319 16:34:01.793666 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:01.793676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:01.793765 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:01.793899 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:01.793900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:01.794059 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:01.794222 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:01.794224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.794951 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:01.794952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:01.795341 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:01.795343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.795430 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:01.795905 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:01.795910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:01.797771 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
W20260319 16:34:01.797967 2730 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:01.793705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:38367 (local address 127.24.187.254:37313)
0319 16:34:01.793791 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:01.793793 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:01.793801 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:34:01.793823 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:01.793824 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:01.793833 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:01.793884 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:01.794318 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.794855 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:01.795451 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.795596 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:01.797900 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:38367: BlockingRecv error: recv got EOF from 127.24.187.194:38367 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Mar 19 16:34:01 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:01 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:01.950105 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:01.944577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:33931)
0319 16:34:01.944731 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:01.944743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:01.944808 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:01.944950 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:01.944951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:01.945126 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:01.945684 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:01.945891 (+ 207us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.946584 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:01.946585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:01.946964 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:01.946966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.947030 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:01.947708 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:01.947713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:01.950024 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
W20260319 16:34:01.950265 2730 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:01.944704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:33931 (local address 127.24.187.254:37313)
0319 16:34:01.944774 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:01.944776 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:01.944792 (+ 16us) server_negotiation.cc:408] Connection header received
0319 16:34:01.944874 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:01.944875 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:01.944886 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:01.944931 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:01.946011 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.946504 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:01.947046 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:01.947210 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:01.950188 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:33931: BlockingRecv error: recv got EOF from 127.24.187.195:33931 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Mar 19 16:34:02 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:02 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:02.678875 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:02.674583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:42421)
0319 16:34:02.674738 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:02.674749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:02.674806 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:02.674948 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:02.674949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:02.675109 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:02.675271 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:02.675273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.675977 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:02.675978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:02.676367 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:02.676368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.676428 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:02.676974 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:02.676978 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:02.678816 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
W20260319 16:34:02.679005 2734 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:02.674710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:42421 (local address 127.24.187.254:37313)
0319 16:34:02.674833 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:02.674835 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:02.674845 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:34:02.674868 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:02.674868 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:02.674878 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:02.674936 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:02.675366 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.675897 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:02.676454 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.676616 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:02.678933 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:42421: BlockingRecv error: recv got EOF from 127.24.187.193:42421 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
Mar 19 16:34:02 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:02 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:02.802683 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:02.798326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:46597)
0319 16:34:02.798495 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:02.798506 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:02.798566 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:02.798810 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:02.798812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:02.798983 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:02.799138 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:02.799140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.799864 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:02.799865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:02.800237 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:02.800239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.800298 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:02.800760 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:02.800764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:02.802601 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":61,"threads_started":1}
W20260319 16:34:02.802798 2734 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:02.798485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:46597 (local address 127.24.187.254:37313)
0319 16:34:02.798543 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:02.798545 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:02.798554 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:34:02.798688 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:02.798689 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:02.798701 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:02.798759 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:02.799230 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.799773 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:02.800400 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.800538 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:02.802744 (+ 2206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:46597: BlockingRecv error: recv got EOF from 127.24.187.194:46597 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Mar 19 16:34:02 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:02 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:02.955133 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:02.950740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:51745)
0319 16:34:02.950910 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:02.950921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:02.950984 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:02.951156 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:02.951157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:02.951325 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:02.951494 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:02.951496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.952188 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:02.952189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:02.952573 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:02.952574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.952643 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:02.953124 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:02.953128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:02.955071 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
W20260319 16:34:02.955264 2734 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:02.950841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:51745 (local address 127.24.187.254:37313)
0319 16:34:02.950894 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:02.950896 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:02.951027 (+ 131us) server_negotiation.cc:408] Connection header received
0319 16:34:02.951056 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:02.951057 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:02.951067 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:02.951109 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:02.951591 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.952093 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:02.952673 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:02.952887 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:02.955190 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:51745: BlockingRecv error: recv got EOF from 127.24.187.195:51745 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Mar 19 16:34:03 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:03 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:03.689862 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:03.679322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:41049)
0319 16:34:03.679458 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:03.679469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:03.679528 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:03.679658 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:03.679659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:03.679839 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:03.679994 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:03.679995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.680709 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:03.680710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:03.681084 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:03.681085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.681140 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:03.687915 (+ 6775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:03.687920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:03.689780 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":117,"threads_started":1}
W20260319 16:34:03.690040 2738 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:03.679368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:41049 (local address 127.24.187.254:37313)
0319 16:34:03.679474 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:03.679477 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:03.679513 (+ 36us) server_negotiation.cc:408] Connection header received
0319 16:34:03.679575 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:03.679576 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:03.679587 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:03.679648 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:03.680065 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.680613 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:03.681173 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.681314 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:03.689900 (+ 8586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:41049: BlockingRecv error: recv got EOF from 127.24.187.193:41049 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
Mar 19 16:34:03 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:03 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:03.807530 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:03.803229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:52467)
0319 16:34:03.803367 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:03.803380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:03.803480 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:03.803605 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:03.803607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:03.803765 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:03.803924 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:03.803926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.804587 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:03.804588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:03.804981 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:03.804982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.805044 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:03.805505 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:03.805509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:03.807471 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
W20260319 16:34:03.807646 2738 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:03.803344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:52467 (local address 127.24.187.254:37313)
0319 16:34:03.803426 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:03.803428 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:03.803461 (+ 33us) server_negotiation.cc:408] Connection header received
0319 16:34:03.803532 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:03.803533 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:03.803546 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:03.803594 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:03.803989 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.804506 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:03.805070 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.805212 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:03.807581 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:52467: BlockingRecv error: recv got EOF from 127.24.187.194:52467 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Mar 19 16:34:03 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:03 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:03.959625 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:03.955697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:56161)
0319 16:34:03.955842 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:03.955854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:03.955916 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:03.956032 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:03.956033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:03.956190 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:03.956344 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:03.956346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.956989 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:03.956990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:03.957351 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:03.957352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.957404 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:03.957880 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:03.957884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:03.959568 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
W20260319 16:34:03.959725 2738 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:03.955796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:56161 (local address 127.24.187.254:37313)
0319 16:34:03.955862 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:03.955864 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:03.955899 (+ 35us) server_negotiation.cc:408] Connection header received
0319 16:34:03.955965 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:03.955965 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:03.955975 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:03.956022 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:03.956422 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.956924 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:03.957431 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:03.957564 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:03.959664 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:56161: BlockingRecv error: recv got EOF from 127.24.187.195:56161 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Mar 19 16:34:04 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:04 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:04.694537 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:04.690340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:57875)
0319 16:34:04.690475 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:04.690485 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:04.690530 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:04.690849 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:04.690851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:04.691021 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:04.691190 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:04.691192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.691910 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:04.691911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:04.692283 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:04.692284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.692340 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:04.692812 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:04.692816 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:04.694481 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
W20260319 16:34:04.694649 2742 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:04.690496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:57875 (local address 127.24.187.254:37313)
0319 16:34:04.690602 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:04.690603 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:04.690611 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:34:04.690632 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:04.690633 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:04.690642 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:04.690741 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:04.691293 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.691822 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:04.692344 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.692458 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:04.694576 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:57875: BlockingRecv error: recv got EOF from 127.24.187.193:57875 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
Mar 19 16:34:04 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:04 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:04.812788 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:04.808016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:37887)
0319 16:34:04.808142 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:04.808152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:04.808201 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:04.808333 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:04.808334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:04.808506 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:04.808674 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:04.808676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.809565 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:04.809567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:04.810062 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:04.810066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.810141 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:04.810638 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:04.810646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:04.812705 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260319 16:34:04.812963 2742 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:04.808096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:37887 (local address 127.24.187.254:37313)
0319 16:34:04.808171 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:04.808173 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:04.808185 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:34:04.808253 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:04.808254 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:04.808266 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:04.808321 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:04.808765 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.809467 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:04.810174 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.810360 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:04.812878 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:37887: BlockingRecv error: recv got EOF from 127.24.187.194:37887 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Mar 19 16:34:04 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:04 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:04.964102 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:04.960083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:55097)
0319 16:34:04.960229 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:04.960242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:04.960315 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:04.960507 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:04.960508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:04.960681 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:04.960830 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:04.960832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.961487 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:04.961489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:04.961889 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:04.961891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.961949 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:04.962402 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:04.962406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:04.964047 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
W20260319 16:34:04.964231 2742 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:04.960168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:55097 (local address 127.24.187.254:37313)
0319 16:34:04.960251 (+ 83us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:04.960253 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:04.960302 (+ 49us) server_negotiation.cc:408] Connection header received
0319 16:34:04.960431 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:04.960432 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:04.960443 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:04.960484 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:04.960901 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.961414 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:04.961954 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:04.962112 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:04.964166 (+ 2054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:55097: BlockingRecv error: recv got EOF from 127.24.187.195:55097 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Mar 19 16:34:05 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:05 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:05.699477 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:05.695063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:50115)
0319 16:34:05.695203 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:05.695213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:05.695265 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:05.695511 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:05.695513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:05.695697 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:05.695861 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:05.695863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.696592 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:05.696593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:05.696977 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:05.696979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.697045 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:05.697644 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:05.697648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:05.699418 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260319 16:34:05.699622 2747 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:05.695191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:50115 (local address 127.24.187.254:37313)
0319 16:34:05.695332 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:05.695334 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:05.695344 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:34:05.695370 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:05.695371 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:05.695382 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:05.695452 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:05.695971 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.696489 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:05.697067 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.697212 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:05.699557 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:50115: BlockingRecv error: recv got EOF from 127.24.187.193:50115 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
Mar 19 16:34:05 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:05 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:05.821031 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:05.813480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:37907)
0319 16:34:05.813631 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:05.813643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:05.813702 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:05.813868 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:05.813870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:05.814043 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:05.814191 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:05.814194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.814848 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:05.814849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:05.815206 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:05.815208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.815266 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:05.819095 (+ 3829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:05.819100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:05.820955 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
W20260319 16:34:05.821127 2747 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:05.813635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:37907 (local address 127.24.187.254:37313)
0319 16:34:05.813699 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:05.813701 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:05.813733 (+ 32us) server_negotiation.cc:408] Connection header received
0319 16:34:05.813797 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:05.813798 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:05.813810 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:05.813853 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:05.814269 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.814771 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:05.815274 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.815450 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:05.821075 (+ 5625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:37907: BlockingRecv error: recv got EOF from 127.24.187.194:37907 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Mar 19 16:34:05 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:05 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:05.968499 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:05.964591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:35769)
0319 16:34:05.964752 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:05.964762 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:05.964821 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:05.964949 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:05.964950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:05.965111 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:05.965272 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:05.965274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.965931 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:05.965933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:05.966307 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:05.966309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.966365 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:05.966856 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:05.966860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:05.968436 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
W20260319 16:34:05.968600 2747 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:05.964687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:35769 (local address 127.24.187.254:37313)
0319 16:34:05.964762 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:05.964764 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:05.964816 (+ 52us) server_negotiation.cc:408] Connection header received
0319 16:34:05.964867 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:05.964867 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:05.964876 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:05.964930 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:05.965327 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.965861 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:05.966376 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:05.966509 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:05.968535 (+ 2026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:35769: BlockingRecv error: recv got EOF from 127.24.187.195:35769 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Mar 19 16:34:06 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:06 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:06.704187 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:06.700003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:54813)
0319 16:34:06.700154 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:06.700164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:06.700208 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:06.700353 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:06.700354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:06.700533 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:06.700693 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:06.700695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.701531 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:06.701532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:06.701937 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:06.701938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.701999 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:06.702550 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:06.702554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:06.704131 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260319 16:34:06.704344 2751 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:06.700114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:54813 (local address 127.24.187.254:37313)
0319 16:34:06.700222 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:06.700224 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:06.700234 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:34:06.700258 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:06.700259 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:06.700268 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:06.700333 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:06.700793 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.701457 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:06.702019 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.702152 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:06.704268 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:54813: BlockingRecv error: recv got EOF from 127.24.187.193:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
Mar 19 16:34:06 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:06 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:06.826004 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:06.821593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:37269)
0319 16:34:06.821871 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:06.821884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:06.821964 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:06.822145 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:06.822146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:06.822399 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:06.822610 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:06.822613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.823319 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:06.823320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:06.823724 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:06.823727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.823791 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:06.824273 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:06.824277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:06.825946 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":70,"threads_started":1}
W20260319 16:34:06.826155 2751 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:06.821680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:37269 (local address 127.24.187.254:37313)
0319 16:34:06.821826 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:06.821828 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:06.821947 (+ 119us) server_negotiation.cc:408] Connection header received
0319 16:34:06.822059 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:06.822060 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:06.822073 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:06.822124 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:06.822703 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.823220 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:06.823813 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.823980 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:06.826088 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:37269: BlockingRecv error: recv got EOF from 127.24.187.194:37269 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Mar 19 16:34:06 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:06 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:06.973238 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:06.969040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:45691)
0319 16:34:06.969248 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:06.969262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:06.969320 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:06.969461 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:06.969462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:06.969620 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:06.969808 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:06.969810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.970561 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:06.970562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:06.970954 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:06.970956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.971004 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:06.971487 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:06.971491 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:06.973170 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
W20260319 16:34:06.973349 2751 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:06.969214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:45691 (local address 127.24.187.254:37313)
0319 16:34:06.969319 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:06.969321 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:06.969356 (+ 35us) server_negotiation.cc:408] Connection header received
0319 16:34:06.969379 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:06.969380 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:06.969389 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:06.969430 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:06.969891 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.970489 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:06.971031 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:06.971186 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:06.973283 (+ 2097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:45691: BlockingRecv error: recv got EOF from 127.24.187.195:45691 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Mar 19 16:34:07 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:07 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:07.708999 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:07.704805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:48565)
0319 16:34:07.704967 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:07.704979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:07.705029 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:07.705201 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:07.705202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:07.705379 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:07.705555 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:07.705557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.706253 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:07.706254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:07.706631 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:07.706632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.706709 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:07.707164 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:07.707168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:07.708942 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
W20260319 16:34:07.709095 2755 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:07.704944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:48565 (local address 127.24.187.254:37313)
0319 16:34:07.705067 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:07.705068 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:07.705078 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:34:07.705103 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:07.705105 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:07.705114 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:07.705191 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:07.705625 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.706169 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:07.706712 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.706840 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:07.709035 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:48565: BlockingRecv error: recv got EOF from 127.24.187.193:48565 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
Mar 19 16:34:07 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:07 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:07.830490 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:07.826444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:44365)
0319 16:34:07.826578 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:07.826589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:07.826658 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:07.826804 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:07.826805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:07.827016 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:07.827169 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:07.827172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.827842 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:07.827843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:07.828222 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:07.828223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.828285 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:07.828769 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:07.828773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:07.830433 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
W20260319 16:34:07.830586 2755 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:07.826532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:44365 (local address 127.24.187.254:37313)
0319 16:34:07.826585 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:07.826587 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:07.826651 (+ 64us) server_negotiation.cc:408] Connection header received
0319 16:34:07.826711 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:07.826712 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:07.826725 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:07.826788 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:07.827238 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.827750 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:07.828335 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.828469 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:07.830527 (+ 2058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:44365: BlockingRecv error: recv got EOF from 127.24.187.194:44365 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Mar 19 16:34:07 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:07 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:07.977484 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:07.973659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:41695)
0319 16:34:07.973821 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:07.973830 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:07.973890 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:07.974022 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:07.974023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:07.974181 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:07.974334 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:07.974336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.974948 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:07.974948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:07.975304 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:07.975306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.975355 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:07.975826 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:07.975831 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:07.977419 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
W20260319 16:34:07.977584 2755 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:07.973768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:41695 (local address 127.24.187.254:37313)
0319 16:34:07.973824 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:07.973826 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:07.973911 (+ 85us) server_negotiation.cc:408] Connection header received
0319 16:34:07.973940 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:07.973941 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:07.973951 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:07.974008 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:07.974403 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.974890 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:07.975369 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:07.975495 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:07.977523 (+ 2028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:41695: BlockingRecv error: recv got EOF from 127.24.187.195:41695 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Mar 19 16:34:08 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:08 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:08.713390 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:08.709526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:38227)
0319 16:34:08.709670 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:08.709683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:08.709763 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:08.709910 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:08.709911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:08.710115 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:08.710272 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:08.710275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.710905 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:08.710906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:08.711263 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:08.711264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.711313 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:08.711751 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:08.711755 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:08.713333 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
W20260319 16:34:08.713517 2759 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:08.709600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:38227 (local address 127.24.187.254:37313)
0319 16:34:08.709739 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:08.709741 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:08.709750 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:34:08.709814 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:08.709815 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:08.709828 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:08.709897 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:08.710339 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.710843 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:08.711332 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.711490 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:08.713452 (+ 1962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:38227: BlockingRecv error: recv got EOF from 127.24.187.193:38227 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
Mar 19 16:34:08 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:08 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:08.835032 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:08.830950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:43149)
0319 16:34:08.831081 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:08.831093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:08.831158 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:08.831264 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:08.831265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:08.831434 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:08.831611 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:08.831614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.832272 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:08.832273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:08.832681 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:08.832683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.832744 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:08.833202 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:08.833206 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:08.834975 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
W20260319 16:34:08.835116 2759 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:08.831045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:43149 (local address 127.24.187.254:37313)
0319 16:34:08.831102 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:08.831104 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:08.831147 (+ 43us) server_negotiation.cc:408] Connection header received
0319 16:34:08.831197 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:08.831198 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:08.831211 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:08.831255 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:08.831699 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.832180 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:08.832768 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.832916 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:08.835059 (+ 2143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:43149: BlockingRecv error: recv got EOF from 127.24.187.194:43149 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Mar 19 16:34:08 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:08 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:08.982157 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:08.977984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:43755)
0319 16:34:08.978123 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:08.978133 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:08.978181 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:08.978350 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:08.978352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:08.978516 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:08.978674 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:08.978676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.979390 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:08.979391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:08.979763 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:08.979765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.979823 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:08.980281 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:08.980285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:08.982100 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
W20260319 16:34:08.982311 2759 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:08.978142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:43755 (local address 127.24.187.254:37313)
0319 16:34:08.978197 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:08.978199 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:08.978207 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:34:08.978228 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:08.978229 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:08.978237 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:08.978293 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:08.978777 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.979271 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:08.979832 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:08.979951 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:08.982245 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:43755: BlockingRecv error: recv got EOF from 127.24.187.195:43755 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Mar 19 16:34:09 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:09 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:09.718015 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:09.713873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:36771)
0319 16:34:09.714090 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:09.714103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:09.714171 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:09.714312 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:09.714313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:09.714510 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:09.714666 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:09.714667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.715297 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:09.715297 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:09.715679 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:09.715681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.715736 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:09.716180 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:09.716184 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:09.717950 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
W20260319 16:34:09.718122 2763 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:09.713957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:36771 (local address 127.24.187.254:37313)
0319 16:34:09.714071 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:09.714073 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:09.714200 (+ 127us) server_negotiation.cc:408] Connection header received
0319 16:34:09.714232 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:09.714233 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:09.714243 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:09.714302 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:09.714733 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.715228 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:09.715759 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.715882 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:09.718059 (+ 2177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:36771: BlockingRecv error: recv got EOF from 127.24.187.193:36771 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
Mar 19 16:34:09 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:09 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:09.839504 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:09.835469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:41619)
0319 16:34:09.835619 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:09.835631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:09.835698 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:09.835844 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:09.835845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:09.836002 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:09.836160 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:09.836162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.836805 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:09.836805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:09.837160 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:09.837161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.837210 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:09.837668 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:09.837678 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:09.839446 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
W20260319 16:34:09.839599 2763 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:09.835585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:41619 (local address 127.24.187.254:37313)
0319 16:34:09.835656 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:09.835658 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:09.835669 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:34:09.835773 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:09.835774 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:09.835785 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:09.835833 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:09.836229 (+ 396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.836741 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:09.837227 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.837359 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:09.839537 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:41619: BlockingRecv error: recv got EOF from 127.24.187.194:41619 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Mar 19 16:34:09 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:09 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:09.987027 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:09.982698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:42329)
0319 16:34:09.982881 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:09.982895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:09.982971 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:09.983103 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:09.983104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:09.983279 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:09.983470 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:09.983472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.984104 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:09.984105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:09.984472 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:09.984474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.984533 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:09.985051 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:09.985056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:09.986969 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
W20260319 16:34:09.987133 2763 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:09.982807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:42329 (local address 127.24.187.254:37313)
0319 16:34:09.982885 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:09.982888 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:09.982956 (+ 68us) server_negotiation.cc:408] Connection header received
0319 16:34:09.983032 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:09.983033 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:09.983044 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:09.983093 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:09.983547 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.984033 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:09.984538 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:09.984706 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:09.987067 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:42329: BlockingRecv error: recv got EOF from 127.24.187.195:42329 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Mar 19 16:34:10 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:10 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:10.723006 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:10.718551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:59021)
0319 16:34:10.718721 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:10.718733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:10.718796 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:10.718954 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:10.718960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:10.719118 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:10.719274 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:10.719275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.719940 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:10.719940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:10.720309 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:10.720311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.720367 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:10.720884 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:10.720889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:10.722925 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
W20260319 16:34:10.723233 2767 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:10.718677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:59021 (local address 127.24.187.254:37313)
0319 16:34:10.718789 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:10.718791 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:10.718800 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:34:10.718846 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:10.718846 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:10.718856 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:10.718912 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:10.719338 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.719867 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:10.720386 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.720549 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:10.723180 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:59021: BlockingRecv error: recv got EOF from 127.24.187.193:59021 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
Mar 19 16:34:10 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:10 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:10.848249 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:10.840021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:34681)
0319 16:34:10.840151 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:10.840161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:10.840227 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:10.840417 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:10.840419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:10.840576 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:10.840734 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:10.840736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.841388 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:10.841389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:10.841819 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:10.841821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.841880 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:10.846553 (+ 4673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:10.846559 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:10.848191 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
W20260319 16:34:10.848489 2767 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:10.840114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:34681 (local address 127.24.187.254:37313)
0319 16:34:10.840180 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:10.840182 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:10.840210 (+ 28us) server_negotiation.cc:408] Connection header received
0319 16:34:10.840288 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:10.840288 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:10.840300 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:10.840358 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:10.840811 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.841291 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:10.841913 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.842057 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:10.848426 (+ 6369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:34681: BlockingRecv error: recv got EOF from 127.24.187.194:34681 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Mar 19 16:34:10 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:10 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.195@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:10.991700 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:10.987504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.195:34493)
0319 16:34:10.987649 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:10.987660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:10.987728 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:10.987867 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:10.987868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:10.988067 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:10.988220 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:10.988222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.988875 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:10.988876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:10.989259 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:10.989261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.989324 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:10.989799 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:10.989803 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:10.991625 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
W20260319 16:34:10.991780 2767 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:10.987543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:34493 (local address 127.24.187.254:37313)
0319 16:34:10.987587 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:10.987589 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:10.987711 (+ 122us) server_negotiation.cc:408] Connection header received
0319 16:34:10.987790 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:10.987791 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:10.987803 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:10.987847 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:10.988286 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.988770 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:10.989341 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:10.989484 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:10.991730 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.195:34493: BlockingRecv error: recv got EOF from 127.24.187.195:34493 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Mar 19 16:34:11 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:11 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.193@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:11.728179 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:11.723500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.193:57025)
0319 16:34:11.723650 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:11.723661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:11.723710 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:11.723820 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:11.723820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:11.723987 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:11.724162 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:11.724164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.725089 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:11.725091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:11.725656 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:11.725660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.725751 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:11.726323 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:11.726328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:11.728122 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
W20260319 16:34:11.728319 2771 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:11.723563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:57025 (local address 127.24.187.254:37313)
0319 16:34:11.723681 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:11.723683 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:11.723694 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:34:11.723754 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:11.723755 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:11.723764 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:11.723816 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:11.724236 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.724990 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:11.725754 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.725933 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:11.728239 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.193:57025: BlockingRecv error: recv got EOF from 127.24.187.193:57025 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":46,"threads_started":1}
Mar 19 16:34:11 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
Mar 19 16:34:11 dist-test-slave-svn8 krb5kdc[1695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.187.194@KRBTEST.COM for oryx/127.24.187.254@KRBTEST.COM, Server not found in Kerberos database
I20260319 16:34:11.853258 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:11.848848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37313 (local address 127.24.187.194:41137)
0319 16:34:11.849038 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:11.849049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:11.849108 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:11.849277 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:11.849278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:11.849445 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:11.849603 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:11.849605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.850471 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:11.850472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:11.850848 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:11.850850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.850911 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:11.851433 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:11.851438 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:11.853199 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.187.254:37313: Server oryx/127.24.187.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
W20260319 16:34:11.853348 2771 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:11.848967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:41137 (local address 127.24.187.254:37313)
0319 16:34:11.849065 (+ 98us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:11.849067 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:11.849093 (+ 26us) server_negotiation.cc:408] Connection header received
0319 16:34:11.849165 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:11.849166 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:11.849178 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:11.849231 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:11.849758 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.850336 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:11.850927 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:11.851058 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:11.853291 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.187.194:41137: BlockingRecv error: recv got EOF from 127.24.187.194:41137 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260319 16:34:11.930449 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2293
I20260319 16:34:11.935079 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2447
I20260319 16:34:11.939110 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2579
I20260319 16:34:11.943006 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2222
2026-03-19T16:34:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18301 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-03-19T16:34:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:34:11Z Disabled control of system clock
I20260319 16:34:11.969395 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36567
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:46541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:36567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260319 16:34:12.046365 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.046532 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.046552 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.048133 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:34:12.048183 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.048195 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.048206 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:34:12.048216 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:34:12.049805 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:36567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.050021 2776 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.050235 2776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:12.052631 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.052673 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.052676 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.052954 2776 server_base.cc:1060] running on GCE node
I20260319 16:34:12.053195 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.053423 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.054569 2776 hybrid_clock.cc:648] HybridClock initialized: now 1773938052054548 us; error 30 us; skew 500 ppm
I20260319 16:34:12.055728 2776 webserver.cc:492] Webserver started at http://127.24.187.254:44681/ using document root <none> and password file <none>
I20260319 16:34:12.055913 2776 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.055954 2776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.056059 2776 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.057044 2776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "a9c9a27384314459a9c394ed2aaa3248"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "37fc9706af001ca1135e6a32699f4a70"
server_key_iv: "052cfcf436cf3bbf1e1d88b0deb2a875"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.057335 2776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "a9c9a27384314459a9c394ed2aaa3248"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "37fc9706af001ca1135e6a32699f4a70"
server_key_iv: "052cfcf436cf3bbf1e1d88b0deb2a875"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.058517 2776 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:34:12.059335 2790 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.059553 2776 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:34:12.059621 2776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "a9c9a27384314459a9c394ed2aaa3248"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "37fc9706af001ca1135e6a32699f4a70"
server_key_iv: "052cfcf436cf3bbf1e1d88b0deb2a875"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.059690 2776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.070003 2776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.070245 2776 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.070348 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.074020 2776 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:36567
I20260319 16:34:12.074048 2842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:36567 every 8 connection(s)
I20260319 16:34:12.074340 2776 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:34:12.074992 2843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:34:12.077294 2843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248: Bootstrap starting.
I20260319 16:34:12.077905 2843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248: Neither blocks nor log segments found. Creating new log.
I20260319 16:34:12.078135 2843 log.cc:826] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248: Log is configured to *not* fsync() on all Append() calls
I20260319 16:34:12.078743 2843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248: No bootstrap required, opened a new log
I20260319 16:34:12.079912 2843 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c9a27384314459a9c394ed2aaa3248" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36567 } }
I20260319 16:34:12.080056 2843 raft_consensus.cc:385] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:34:12.080087 2843 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c9a27384314459a9c394ed2aaa3248, State: Initialized, Role: FOLLOWER
I20260319 16:34:12.080199 2843 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c9a27384314459a9c394ed2aaa3248" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36567 } }
I20260319 16:34:12.080260 2843 raft_consensus.cc:399] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:34:12.080287 2843 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:34:12.080330 2843 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:34:12.080977 2843 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c9a27384314459a9c394ed2aaa3248" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36567 } }
I20260319 16:34:12.081081 2843 leader_election.cc:304] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9c9a27384314459a9c394ed2aaa3248; no voters:
I20260319 16:34:12.081199 2843 leader_election.cc:290] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:34:12.081254 2846 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:34:12.081388 2843 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:34:12.081432 2846 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [term 1 LEADER]: Becoming Leader. State: Replica: a9c9a27384314459a9c394ed2aaa3248, State: Running, Role: LEADER
I20260319 16:34:12.081511 2846 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c9a27384314459a9c394ed2aaa3248" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36567 } }
I20260319 16:34:12.081895 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9c9a27384314459a9c394ed2aaa3248. Latest consensus state: current_term: 1 leader_uuid: "a9c9a27384314459a9c394ed2aaa3248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c9a27384314459a9c394ed2aaa3248" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36567 } } }
I20260319 16:34:12.081923 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9c9a27384314459a9c394ed2aaa3248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9c9a27384314459a9c394ed2aaa3248" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36567 } } }
I20260319 16:34:12.082005 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [sys.catalog]: This master's current role is: LEADER
I20260319 16:34:12.082012 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248 [sys.catalog]: This master's current role is: LEADER
I20260319 16:34:12.082191 2854 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:34:12.082610 2854 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:34:12.083627 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2776
I20260319 16:34:12.083725 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:34:12.083904 25327 external_mini_cluster.cc:1468] Setting key 1dd6bd2c852a368b3974401843b5605a
I20260319 16:34:12.083966 2854 catalog_manager.cc:1357] Generated new cluster ID: 83c5c206ff264454854b413c7cd00b0e
I20260319 16:34:12.084002 2854 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:34:12.088609 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.084349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.24.187.254:36567)
0319 16:34:12.084623 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.084629 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.084650 (+ 21us) server_negotiation.cc:408] Connection header received
0319 16:34:12.084751 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.084757 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.084826 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.084953 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.085375 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.086501 (+ 1126us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:12.087384 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.087601 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:12.087986 (+ 385us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.088014 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.088014 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.088227 (+ 213us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.088261 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.088269 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.088331 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.088407 (+ 76us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.088460 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
I20260319 16:34:12.098874 2854 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:34:12.099462 2854 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:34:12.105113 2854 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a9c9a27384314459a9c394ed2aaa3248: Generated new TSK 0
I20260319 16:34:12.105269 2854 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:34:12.109097 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36567
--builtin_ntp_servers=127.24.187.212:46541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260319 16:34:12.186757 2867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.186932 2867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.186954 2867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.188771 2867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.188832 2867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.188858 2867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:34:12.190608 2867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.190831 2867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.191053 2867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:12.193699 2875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.193825 2867 server_base.cc:1060] running on GCE node
W20260319 16:34:12.193910 2872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.193703 2873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.194159 2867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.194388 2867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.195541 2867 hybrid_clock.cc:648] HybridClock initialized: now 1773938052195524 us; error 29 us; skew 500 ppm
I20260319 16:34:12.196663 2867 webserver.cc:492] Webserver started at http://127.24.187.193:37235/ using document root <none> and password file <none>
I20260319 16:34:12.196837 2867 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.196878 2867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.196981 2867 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.197968 2867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "d4ea7732e6fd4b2d944355996a8b7659"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "248f1bc3145cdb35bebc4a29324151d5"
server_key_iv: "4d59344f933f20d228556de85b8dc36f"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.198272 2867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "d4ea7732e6fd4b2d944355996a8b7659"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "248f1bc3145cdb35bebc4a29324151d5"
server_key_iv: "4d59344f933f20d228556de85b8dc36f"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.199465 2867 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:34:12.200268 2881 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.200598 2867 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:34:12.200686 2867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "d4ea7732e6fd4b2d944355996a8b7659"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "248f1bc3145cdb35bebc4a29324151d5"
server_key_iv: "4d59344f933f20d228556de85b8dc36f"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.200757 2867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.213968 2867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.214188 2867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.214288 2867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.214497 2867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:12.214792 2867 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:12.214824 2867 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.214854 2867 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:12.214877 2867 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.220323 2867 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:40137
I20260319 16:34:12.220388 2994 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:40137 every 8 connection(s)
I20260319 16:34:12.220679 2867 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:34:12.224043 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2867
I20260319 16:34:12.224145 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:34:12.224319 25327 external_mini_cluster.cc:1468] Setting key 0ea531e93e76f11f94966003186b7bff
I20260319 16:34:12.225306 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36567
--builtin_ntp_servers=127.24.187.212:46541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260319 16:34:12.226303 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.221398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:37495 (local address 127.24.187.254:36567)
0319 16:34:12.221460 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.221463 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.221657 (+ 194us) server_negotiation.cc:408] Connection header received
0319 16:34:12.222350 (+ 693us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.222353 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.222368 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.222425 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.223024 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.223550 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:12.224306 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.224504 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:12.226036 (+ 1532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.226040 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.226040 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.226045 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.226059 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.226063 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.226108 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.226192 (+ 84us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.226229 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:34:12.226476 2997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.221349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36567 (local address 127.24.187.193:37495)
0319 16:34:12.221543 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:12.221576 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:12.222225 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:12.222495 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:12.222500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:12.222510 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:12.222932 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:12.222936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.223636 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:12.223637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:12.224222 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:12.224225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.224370 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:12.225845 (+ 1475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:34:12.225845 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:12.225912 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:34:12.225934 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:12.226114 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:12.226119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:12.226121 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:34:12.226192 (+ 71us) client_negotiation.cc:240] Negotiation successful
0319 16:34:12.226271 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260319 16:34:12.227240 2995 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36567
I20260319 16:34:12.227331 2995 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:12.227530 2995 heartbeater.cc:507] Master 127.24.187.254:36567 requested a full tablet report, sending...
I20260319 16:34:12.228130 2807 ts_manager.cc:194] Registered new tserver with Master: d4ea7732e6fd4b2d944355996a8b7659 (127.24.187.193:40137)
I20260319 16:34:12.228799 2807 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.193:37495
W20260319 16:34:12.308672 2998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.308830 2998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.308849 2998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.310284 2998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.310321 2998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.310343 2998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:34:12.311841 2998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.312012 2998 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.312175 2998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:12.314956 3004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.314939 3003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.315061 2998 server_base.cc:1060] running on GCE node
W20260319 16:34:12.314929 3006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.315374 2998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.315588 2998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.316740 2998 hybrid_clock.cc:648] HybridClock initialized: now 1773938052316722 us; error 34 us; skew 500 ppm
I20260319 16:34:12.317878 2998 webserver.cc:492] Webserver started at http://127.24.187.194:32843/ using document root <none> and password file <none>
I20260319 16:34:12.318053 2998 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.318097 2998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.318202 2998 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.319103 2998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "5dc2629984564c688d7e8ebf43db93ad"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "a6ee646dc415568bad5c03fb0d5dd34e"
server_key_iv: "fa16e07318f5caad19d84f686962f357"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.319387 2998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "5dc2629984564c688d7e8ebf43db93ad"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "a6ee646dc415568bad5c03fb0d5dd34e"
server_key_iv: "fa16e07318f5caad19d84f686962f357"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.320492 2998 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:34:12.321175 3012 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.321343 2998 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:34:12.321408 2998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "5dc2629984564c688d7e8ebf43db93ad"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "a6ee646dc415568bad5c03fb0d5dd34e"
server_key_iv: "fa16e07318f5caad19d84f686962f357"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.321465 2998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.331054 2998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.331271 2998 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.331372 2998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.331598 2998 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:12.331890 2998 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:12.331923 2998 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.331950 2998 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:12.331974 2998 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.337618 2998 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:34045
I20260319 16:34:12.337677 3126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:34045 every 8 connection(s)
I20260319 16:34:12.337987 2998 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:34:12.341074 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 2998
I20260319 16:34:12.341157 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:34:12.341343 25327 external_mini_cluster.cc:1468] Setting key 8cc44e47ee3f7ca1877629d12777f964
I20260319 16:34:12.342453 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.338750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:41601 (local address 127.24.187.254:36567)
0319 16:34:12.338815 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.338817 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.339018 (+ 201us) server_negotiation.cc:408] Connection header received
0319 16:34:12.339525 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.339527 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.339547 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.339673 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.340198 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.340691 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:12.341271 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.341442 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:12.342041 (+ 599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.342044 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.342045 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.342050 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.342065 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.342069 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.342121 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.342286 (+ 165us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.342367 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:34:12.342487 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36567
--builtin_ntp_servers=127.24.187.212:46541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260319 16:34:12.342556 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.338699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36567 (local address 127.24.187.194:41601)
0319 16:34:12.338914 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:12.338938 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:12.339429 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:12.339765 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:12.339769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:12.339778 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:12.340084 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:12.340086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.340772 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:12.340773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:12.341188 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:12.341191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.341337 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:12.341921 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:34:12.341921 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:12.341948 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:34:12.341970 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:12.342151 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:12.342158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:12.342162 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:34:12.342256 (+ 94us) client_negotiation.cc:240] Negotiation successful
0319 16:34:12.342391 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260319 16:34:12.342978 3127 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36567
I20260319 16:34:12.343070 3127 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:12.343266 3127 heartbeater.cc:507] Master 127.24.187.254:36567 requested a full tablet report, sending...
I20260319 16:34:12.343669 2807 ts_manager.cc:194] Registered new tserver with Master: 5dc2629984564c688d7e8ebf43db93ad (127.24.187.194:34045)
I20260319 16:34:12.344053 2807 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.194:41601
W20260319 16:34:12.418077 3130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.418241 3130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.418260 3130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.419785 3130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.419827 3130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.419850 3130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:34:12.421371 3130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.421567 3130 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.421792 3130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:12.424242 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.424237 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.424239 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.424530 3130 server_base.cc:1060] running on GCE node
I20260319 16:34:12.424714 3130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.424927 3130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.426151 3130 hybrid_clock.cc:648] HybridClock initialized: now 1773938052426134 us; error 23 us; skew 500 ppm
I20260319 16:34:12.427325 3130 webserver.cc:492] Webserver started at http://127.24.187.195:39779/ using document root <none> and password file <none>
I20260319 16:34:12.427523 3130 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.427563 3130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.427664 3130 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.428561 3130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "febf2acbb89a4bb0bda5223375aa0b08"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "08981b84836d1417093d3a2e7f2640d0"
server_key_iv: "b39a3228a6f4a79b6f9a790ae06e7be3"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.428844 3130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "febf2acbb89a4bb0bda5223375aa0b08"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "08981b84836d1417093d3a2e7f2640d0"
server_key_iv: "b39a3228a6f4a79b6f9a790ae06e7be3"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.429944 3130 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:34:12.430626 3144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.430785 3130 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:34:12.430841 3130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "febf2acbb89a4bb0bda5223375aa0b08"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "08981b84836d1417093d3a2e7f2640d0"
server_key_iv: "b39a3228a6f4a79b6f9a790ae06e7be3"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.430902 3130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.449857 3130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.450109 3130 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.450218 3130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.450423 3130 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:12.450742 3130 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:12.450775 3130 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.450805 3130 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:12.450822 3130 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.455947 3130 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:35397
I20260319 16:34:12.456024 3257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:35397 every 8 connection(s)
I20260319 16:34:12.456267 3130 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:34:12.457767 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 3130
I20260319 16:34:12.457868 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:34:12.458042 25327 external_mini_cluster.cc:1468] Setting key 22b231aea9473e3d23171004550c6afa
I20260319 16:34:12.460552 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.456908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:33315 (local address 127.24.187.254:36567)
0319 16:34:12.456978 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.456981 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.457147 (+ 166us) server_negotiation.cc:408] Connection header received
0319 16:34:12.457695 (+ 548us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.457697 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.457731 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.457796 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.458372 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.458921 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:12.459452 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.459613 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:12.460278 (+ 665us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.460282 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.460283 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.460289 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.460313 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.460316 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.460360 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.460440 (+ 80us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.460475 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:34:12.460665 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.456879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36567 (local address 127.24.187.195:33315)
0319 16:34:12.457040 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:12.457066 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:12.457568 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:12.457857 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:12.457862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:12.457873 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:12.458271 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:12.458275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.459009 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:12.459009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:12.459382 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:12.459384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:12.459516 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:12.460147 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:34:12.460148 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:12.460167 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:34:12.460185 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:12.460373 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:12.460377 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:12.460380 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:34:12.460440 (+ 60us) client_negotiation.cc:240] Negotiation successful
0319 16:34:12.460514 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260319 16:34:12.461081 3258 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36567
I20260319 16:34:12.461174 3258 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:12.461354 3258 heartbeater.cc:507] Master 127.24.187.254:36567 requested a full tablet report, sending...
I20260319 16:34:12.461814 2807 ts_manager.cc:194] Registered new tserver with Master: febf2acbb89a4bb0bda5223375aa0b08 (127.24.187.195:35397)
I20260319 16:34:12.462177 2807 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.195:33315
I20260319 16:34:12.470019 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260319 16:34:12.471935 2866 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:12.471448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.24.187.254:36567)
0319 16:34:12.471502 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.471504 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.471567 (+ 63us) server_negotiation.cc:408] Connection header received
0319 16:34:12.471676 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.471678 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.471687 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.471731 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.471868 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40326: BlockingRecv error: recv got EOF from 127.0.0.1:40326 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:34:12.472041 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2867
I20260319 16:34:12.476926 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2998
I20260319 16:34:12.481045 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 3130
I20260319 16:34:12.485152 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 2776
2026-03-19T16:34:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (541 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-03-19T16:34:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:34:12Z Disabled control of system clock
I20260319 16:34:12.510157 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42263
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260319 16:34:12.587268 3269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.587437 3269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.587455 3269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.588824 3269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:34:12.588860 3269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.588871 3269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.588881 3269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:34:12.588892 3269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:34:12.590422 3269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:42263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.590603 3269 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.590770 3269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:12.593066 3274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.593171 3269 server_base.cc:1060] running on GCE node
W20260319 16:34:12.593071 3275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.593297 3277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.593532 3269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.593781 3269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.594920 3269 hybrid_clock.cc:648] HybridClock initialized: now 1773938052594890 us; error 48 us; skew 500 ppm
I20260319 16:34:12.596001 3269 webserver.cc:492] Webserver started at http://127.24.187.254:44743/ using document root <none> and password file <none>
I20260319 16:34:12.596170 3269 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.596210 3269 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.596312 3269 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.597226 3269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "a3aba5e51edc425da77e1ccbfce4f63f"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "a7b501be42d1732f8d5533945d64178c"
server_key_iv: "12ffba9e11cebe2c59cd79efeee7892a"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.597535 3269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "a3aba5e51edc425da77e1ccbfce4f63f"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "a7b501be42d1732f8d5533945d64178c"
server_key_iv: "12ffba9e11cebe2c59cd79efeee7892a"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.598613 3269 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:34:12.599236 3284 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.599468 3269 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:34:12.599538 3269 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "a3aba5e51edc425da77e1ccbfce4f63f"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "a7b501be42d1732f8d5533945d64178c"
server_key_iv: "12ffba9e11cebe2c59cd79efeee7892a"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.599610 3269 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.603027 3269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.603237 3269 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.603353 3269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.607179 3269 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:42263
I20260319 16:34:12.607206 3336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:42263 every 8 connection(s)
I20260319 16:34:12.607491 3269 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:34:12.608009 3337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:34:12.610280 3337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f: Bootstrap starting.
I20260319 16:34:12.610802 3337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f: Neither blocks nor log segments found. Creating new log.
I20260319 16:34:12.610985 3337 log.cc:826] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f: Log is configured to *not* fsync() on all Append() calls
I20260319 16:34:12.611536 3337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f: No bootstrap required, opened a new log
I20260319 16:34:12.612711 3337 raft_consensus.cc:359] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42263 } }
I20260319 16:34:12.612821 3337 raft_consensus.cc:385] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:34:12.612840 3337 raft_consensus.cc:740] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3aba5e51edc425da77e1ccbfce4f63f, State: Initialized, Role: FOLLOWER
I20260319 16:34:12.612926 3337 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42263 } }
I20260319 16:34:12.612987 3337 raft_consensus.cc:399] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:34:12.613009 3337 raft_consensus.cc:493] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:34:12.613039 3337 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:34:12.613528 3337 raft_consensus.cc:515] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42263 } }
I20260319 16:34:12.613641 3337 leader_election.cc:304] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3aba5e51edc425da77e1ccbfce4f63f; no voters:
I20260319 16:34:12.613819 3337 leader_election.cc:290] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:34:12.613894 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 3269
I20260319 16:34:12.613937 3340 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:34:12.614015 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:34:12.614046 3340 raft_consensus.cc:697] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [term 1 LEADER]: Becoming Leader. State: Replica: a3aba5e51edc425da77e1ccbfce4f63f, State: Running, Role: LEADER
I20260319 16:34:12.614126 3340 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42263 } }
I20260319 16:34:12.614157 3337 sys_catalog.cc:565] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:34:12.614189 25327 external_mini_cluster.cc:1468] Setting key 8d9f2b9468fb5905a77f19be774e3da6
I20260319 16:34:12.615043 3342 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42263 } } }
I20260319 16:34:12.615131 3342 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [sys.catalog]: This master's current role is: LEADER
I20260319 16:34:12.615463 3341 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3aba5e51edc425da77e1ccbfce4f63f. Latest consensus state: current_term: 1 leader_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aba5e51edc425da77e1ccbfce4f63f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42263 } } }
I20260319 16:34:12.615813 3341 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f [sys.catalog]: This master's current role is: LEADER
I20260319 16:34:12.615599 3352 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:34:12.616263 3352 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:34:12.616511 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.614665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.24.187.254:42263)
0319 16:34:12.614859 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.614866 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.614889 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:34:12.614989 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.614995 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.615033 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.615278 (+ 245us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.615990 (+ 712us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.615992 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.615993 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.615996 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.616023 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.616029 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.616103 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.616325 (+ 222us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.616375 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260319 16:34:12.617532 3352 catalog_manager.cc:1357] Generated new cluster ID: 5d374e9a8afa4a498fc87ca42b2bda6d
I20260319 16:34:12.617573 3352 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:34:12.621274 3352 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:34:12.621403 3352 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:34:12.631842 3352 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a3aba5e51edc425da77e1ccbfce4f63f: Generated new TSK 0
I20260319 16:34:12.632014 3352 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:34:12.637161 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42263
--builtin_ntp_servers=127.24.187.212:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260319 16:34:12.745147 3361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.745373 3361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.745410 3361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.747659 3361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.747730 3361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.747771 3361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:34:12.750185 3361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.750427 3361 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.750700 3361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:12.753628 3367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.753648 3361 server_base.cc:1060] running on GCE node
W20260319 16:34:12.753666 3369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.753836 3366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.754072 3361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.754316 3361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.755486 3361 hybrid_clock.cc:648] HybridClock initialized: now 1773938052755466 us; error 30 us; skew 500 ppm
I20260319 16:34:12.756827 3361 webserver.cc:492] Webserver started at http://127.24.187.193:36163/ using document root <none> and password file <none>
I20260319 16:34:12.757035 3361 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.757094 3361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.757202 3361 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.758476 3361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "44b9ab0d477647319b04e3f092f9f419"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "3ecc773a30bcabd0ba41747fe0b659a2"
server_key_iv: "635848fa8c478f1aaa24c597111dc8d6"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.758891 3361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "44b9ab0d477647319b04e3f092f9f419"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "3ecc773a30bcabd0ba41747fe0b659a2"
server_key_iv: "635848fa8c478f1aaa24c597111dc8d6"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.760409 3361 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:34:12.761335 3376 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.761523 3361 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:34:12.761574 3361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "44b9ab0d477647319b04e3f092f9f419"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "3ecc773a30bcabd0ba41747fe0b659a2"
server_key_iv: "635848fa8c478f1aaa24c597111dc8d6"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.761643 3361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.764979 3361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.765173 3361 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.765266 3361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.765450 3361 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:12.765748 3361 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:12.765787 3361 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.765813 3361 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:12.765836 3361 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.770946 3361 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:41063
I20260319 16:34:12.771059 3489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:41063 every 8 connection(s)
I20260319 16:34:12.771288 3361 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:34:12.772954 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 3361
I20260319 16:34:12.773074 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:34:12.773275 25327 external_mini_cluster.cc:1468] Setting key 14e65d101a9681fa906b5e55ca9c7388
I20260319 16:34:12.773901 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.771996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:42405 (local address 127.24.187.254:42263)
0319 16:34:12.772045 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.772048 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.772309 (+ 261us) server_negotiation.cc:408] Connection header received
0319 16:34:12.772779 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.772781 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.772788 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.772837 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.773558 (+ 721us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.773560 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.773560 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.773573 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.773591 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.773595 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.773643 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.773812 (+ 169us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.773828 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:34:12.774250 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42263
--builtin_ntp_servers=127.24.187.212:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260319 16:34:12.776044 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.771964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42263 (local address 127.24.187.193:42405)
0319 16:34:12.772184 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:12.772211 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:12.772669 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:12.772894 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:12.772897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:12.772905 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:12.773396 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:34:12.773396 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:12.773420 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:34:12.773443 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:12.773664 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:12.773669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:12.773672 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:34:12.773811 (+ 139us) client_negotiation.cc:240] Negotiation successful
0319 16:34:12.773937 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260319 16:34:12.776564 3490 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42263
I20260319 16:34:12.776661 3490 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:12.776782 3490 heartbeater.cc:507] Master 127.24.187.254:42263 requested a full tablet report, sending...
I20260319 16:34:12.777283 3301 ts_manager.cc:194] Registered new tserver with Master: 44b9ab0d477647319b04e3f092f9f419 (127.24.187.193:41063)
W20260319 16:34:12.854200 3493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.854359 3493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.854398 3493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.855863 3493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.855906 3493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.855939 3493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:34:12.857439 3493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.857647 3493 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.857872 3493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:12.860634 3498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.860615 3499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.860651 3501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.860950 3493 server_base.cc:1060] running on GCE node
I20260319 16:34:12.861097 3493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.861294 3493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.862442 3493 hybrid_clock.cc:648] HybridClock initialized: now 1773938052862426 us; error 27 us; skew 500 ppm
I20260319 16:34:12.863503 3493 webserver.cc:492] Webserver started at http://127.24.187.194:34709/ using document root <none> and password file <none>
I20260319 16:34:12.863694 3493 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.863736 3493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.863837 3493 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.864737 3493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "44dc92eb488b424fb2715507749e240d"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "87669d5186073b96eef86fb8d148a43d"
server_key_iv: "f624d8b7cd648532a8ed09a24e6e3d93"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.865027 3493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "44dc92eb488b424fb2715507749e240d"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "87669d5186073b96eef86fb8d148a43d"
server_key_iv: "f624d8b7cd648532a8ed09a24e6e3d93"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.866284 3493 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:34:12.867081 3508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.867254 3493 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:34:12.867318 3493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "44dc92eb488b424fb2715507749e240d"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "87669d5186073b96eef86fb8d148a43d"
server_key_iv: "f624d8b7cd648532a8ed09a24e6e3d93"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.867381 3493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.870707 3493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.870891 3493 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.870986 3493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.871241 3493 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:12.871582 3493 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:12.871618 3493 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.871644 3493 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:12.871659 3493 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.876894 3493 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:33345
I20260319 16:34:12.876951 3621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:33345 every 8 connection(s)
I20260319 16:34:12.877199 3493 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:34:12.879355 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 3493
I20260319 16:34:12.879472 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:34:12.879572 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.877853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:53535 (local address 127.24.187.254:42263)
0319 16:34:12.877906 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.877908 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.878091 (+ 183us) server_negotiation.cc:408] Connection header received
0319 16:34:12.878549 (+ 458us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.878550 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.878559 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.878629 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.879287 (+ 658us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.879289 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.879289 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.879294 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.879313 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.879317 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.879366 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.879480 (+ 114us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.879504 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:34:12.879673 25327 external_mini_cluster.cc:1468] Setting key ad4cb77bac2d11bcc4d24592fb628e17
I20260319 16:34:12.879669 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.877829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42263 (local address 127.24.187.194:53535)
0319 16:34:12.877999 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:12.878029 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:12.878452 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:12.878681 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:12.878684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:12.878692 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:12.879160 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:34:12.879160 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:12.879174 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:34:12.879207 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:12.879369 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:12.879373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:12.879376 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:34:12.879445 (+ 69us) client_negotiation.cc:240] Negotiation successful
0319 16:34:12.879495 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260319 16:34:12.880131 3622 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42263
I20260319 16:34:12.880206 3622 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:12.880313 3622 heartbeater.cc:507] Master 127.24.187.254:42263 requested a full tablet report, sending...
I20260319 16:34:12.880627 3300 ts_manager.cc:194] Registered new tserver with Master: 44dc92eb488b424fb2715507749e240d (127.24.187.194:33345)
I20260319 16:34:12.880869 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42263
--builtin_ntp_servers=127.24.187.212:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260319 16:34:12.953661 3625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:12.953858 3625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:12.953879 3625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:12.955418 3625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:12.955469 3625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:12.955494 3625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:34:12.957022 3625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:12.957235 3625 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:12.957451 3625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:34:12.960258 3625 server_base.cc:1060] running on GCE node
W20260319 16:34:12.960258 3630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.960322 3631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:12.960322 3633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:12.960702 3625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:12.960911 3625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:12.962067 3625 hybrid_clock.cc:648] HybridClock initialized: now 1773938052962049 us; error 27 us; skew 500 ppm
I20260319 16:34:12.963182 3625 webserver.cc:492] Webserver started at http://127.24.187.195:40113/ using document root <none> and password file <none>
I20260319 16:34:12.963362 3625 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:12.963404 3625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:12.963507 3625 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:12.964452 3625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "7b12ec58b9404960a145cb3a6842e5af"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "2d491ed1ff75d5a3f792d7b39cc35efb"
server_key_iv: "6b2e6134ffe00e944ffd0db7204426b7"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.964766 3625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "7b12ec58b9404960a145cb3a6842e5af"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "2d491ed1ff75d5a3f792d7b39cc35efb"
server_key_iv: "6b2e6134ffe00e944ffd0db7204426b7"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.965942 3625 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:34:12.966811 3640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.967092 3625 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:34:12.967154 3625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "7b12ec58b9404960a145cb3a6842e5af"
format_stamp: "Formatted at 2026-03-19 16:34:12 on dist-test-slave-svn8"
server_key: "2d491ed1ff75d5a3f792d7b39cc35efb"
server_key_iv: "6b2e6134ffe00e944ffd0db7204426b7"
server_key_version: "encryptionkey@0"
I20260319 16:34:12.967204 3625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:12.970624 3625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:12.970824 3625 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:12.970911 3625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:12.971042 3625 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:12.971267 3625 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:12.971293 3625 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.971318 3625 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:12.971333 3625 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:12.976771 3625 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:43979
I20260319 16:34:12.976846 3753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:43979 every 8 connection(s)
I20260319 16:34:12.977075 3625 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:34:12.979666 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.977788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:42163 (local address 127.24.187.254:42263)
0319 16:34:12.977841 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.977842 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.978038 (+ 196us) server_negotiation.cc:408] Connection header received
0319 16:34:12.978514 (+ 476us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.978515 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.978523 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.978590 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.979399 (+ 809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:12.979401 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:12.979401 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:34:12.979405 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:12.979421 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:34:12.979425 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:12.979481 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:12.979588 (+ 107us) server_negotiation.cc:300] Negotiation successful
0319 16:34:12.979609 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:34:12.979794 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:12.977735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42263 (local address 127.24.187.195:42163)
0319 16:34:12.977907 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:12.977937 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:12.978375 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:12.978681 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:12.978685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:12.978694 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:12.979226 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:34:12.979226 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:12.979246 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:34:12.979267 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:12.979511 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:12.979516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:12.979519 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:34:12.979592 (+ 73us) client_negotiation.cc:240] Negotiation successful
0319 16:34:12.979633 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260319 16:34:12.980381 3754 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42263
I20260319 16:34:12.980466 3754 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:12.980584 3754 heartbeater.cc:507] Master 127.24.187.254:42263 requested a full tablet report, sending...
I20260319 16:34:12.980919 3300 ts_manager.cc:194] Registered new tserver with Master: 7b12ec58b9404960a145cb3a6842e5af (127.24.187.195:43979)
I20260319 16:34:12.985195 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 3625
I20260319 16:34:12.985288 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:34:12.985428 25327 external_mini_cluster.cc:1468] Setting key 076334fbd55fff89ddb8fd99b6e974d1
I20260319 16:34:12.986400 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260319 16:34:12.988157 3346 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:34:12.987685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.24.187.254:42263)
0319 16:34:12.987734 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:12.987736 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:12.987801 (+ 65us) server_negotiation.cc:408] Connection header received
0319 16:34:12.987852 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:12.987852 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:12.987859 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:12.987921 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:12.988098 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47970: BlockingRecv error: recv got EOF from 127.0.0.1:47970 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:34:12.988242 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 3361
I20260319 16:34:12.993052 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 3493
I20260319 16:34:12.997294 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 3625
I20260319 16:34:13.001791 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 3269
2026-03-19T16:34:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (517 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:34:13 dist-test-slave-svn8 krb5kdc[3764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:34:13 dist-test-slave-svn8 krb5kdc[3764](info): set up 2 sockets
Mar 19 16:34:13 dist-test-slave-svn8 krb5kdc[3764](info): commencing operation
krb5kdc: starting...
W20260319 16:34:14.855036 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.834s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:34:14 dist-test-slave-svn8 krb5kdc[3764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:34:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:34:14.992506 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39411
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:34:15.067181 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:15.067373 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:15.067394 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:15.068971 3780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:34:15.069020 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:15.069033 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:15.069044 3780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:34:15.069056 3780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:34:15.070757 3780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:15.070962 3780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:15.071125 3780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:15.073612 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:15.073591 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:15.073688 3780 server_base.cc:1060] running on GCE node
W20260319 16:34:15.073590 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:15.074136 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:15.074361 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:15.075495 3780 hybrid_clock.cc:648] HybridClock initialized: now 1773938055075472 us; error 38 us; skew 500 ppm
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938055, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:34:15.077911 3780 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:34:15.078392 3780 webserver.cc:492] Webserver started at http://127.24.187.254:34397/ using document root <none> and password file <none>
I20260319 16:34:15.078552 3780 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:15.078585 3780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:15.078691 3780 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:15.079618 3780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "27fa7e491bfe4513877ea02912cfd70d"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "aaf8e5c8b5351c4928ec6f911da1ab15"
server_key_iv: "0200dd033f5cd230cae357a6cbcfaf43"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.079914 3780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "27fa7e491bfe4513877ea02912cfd70d"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "aaf8e5c8b5351c4928ec6f911da1ab15"
server_key_iv: "0200dd033f5cd230cae357a6cbcfaf43"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.080973 3780 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:34:15.081655 3795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.081866 3780 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:34:15.081935 3780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "27fa7e491bfe4513877ea02912cfd70d"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "aaf8e5c8b5351c4928ec6f911da1ab15"
server_key_iv: "0200dd033f5cd230cae357a6cbcfaf43"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.082003 3780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:15.088668 3780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:15.090457 3780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:15.090585 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:15.094491 3780 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39411
I20260319 16:34:15.094535 3847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39411 every 8 connection(s)
I20260319 16:34:15.094806 3780 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:34:15.095345 3848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:34:15.097009 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 3780
I20260319 16:34:15.097113 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:34:15.097280 25327 external_mini_cluster.cc:1468] Setting key 80d2cfe29f1f366302c645bb378b813f
I20260319 16:34:15.097818 3848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d: Bootstrap starting.
I20260319 16:34:15.098415 3848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d: Neither blocks nor log segments found. Creating new log.
I20260319 16:34:15.098747 3848 log.cc:826] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d: Log is configured to *not* fsync() on all Append() calls
I20260319 16:34:15.099409 3848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d: No bootstrap required, opened a new log
I20260319 16:34:15.100660 3848 raft_consensus.cc:359] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fa7e491bfe4513877ea02912cfd70d" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39411 } }
I20260319 16:34:15.100807 3848 raft_consensus.cc:385] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:34:15.100838 3848 raft_consensus.cc:740] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fa7e491bfe4513877ea02912cfd70d, State: Initialized, Role: FOLLOWER
I20260319 16:34:15.101176 3848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fa7e491bfe4513877ea02912cfd70d" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39411 } }
I20260319 16:34:15.101289 3848 raft_consensus.cc:399] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:34:15.101320 3848 raft_consensus.cc:493] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:34:15.101358 3848 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:34:15.102093 3848 raft_consensus.cc:515] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fa7e491bfe4513877ea02912cfd70d" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39411 } }
I20260319 16:34:15.102198 3848 leader_election.cc:304] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27fa7e491bfe4513877ea02912cfd70d; no voters:
I20260319 16:34:15.102335 3848 leader_election.cc:290] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:34:15.102375 3853 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:34:15.102505 3853 raft_consensus.cc:697] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [term 1 LEADER]: Becoming Leader. State: Replica: 27fa7e491bfe4513877ea02912cfd70d, State: Running, Role: LEADER
I20260319 16:34:15.102566 3848 sys_catalog.cc:565] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:34:15.102589 3853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fa7e491bfe4513877ea02912cfd70d" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39411 } }
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:34:15.102872 3853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27fa7e491bfe4513877ea02912cfd70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fa7e491bfe4513877ea02912cfd70d" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39411 } } }
I20260319 16:34:15.102949 3853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [sys.catalog]: This master's current role is: LEADER
I20260319 16:34:15.103065 3853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27fa7e491bfe4513877ea02912cfd70d. Latest consensus state: current_term: 1 leader_uuid: "27fa7e491bfe4513877ea02912cfd70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fa7e491bfe4513877ea02912cfd70d" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39411 } } }
I20260319 16:34:15.103117 3853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d [sys.catalog]: This master's current role is: LEADER
I20260319 16:34:15.103221 3858 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:34:15.103860 3858 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:34:15.105039 3870 mvcc.cc:204] Tried to move back new op lower bound from 7266050273708503040 to 7266050273700589568. Current Snapshot: MvccSnapshot[applied={T|T < 7266050273708503040}]
I20260319 16:34:15.105739 3858 catalog_manager.cc:1357] Generated new cluster ID: 6e4ae7c76a4c4f7a80a56855a7e13c3b
I20260319 16:34:15.105782 3858 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:34:15.107043 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.097770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.24.187.254:39411)
0319 16:34:15.097918 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.097928 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.097946 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:34:15.098234 (+ 288us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.098239 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.098274 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.098367 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:15.099227 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.099972 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.100736 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.100983 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.103289 (+ 2306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:15.103296 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:15.103296 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:34:15.103311 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:15.105194 (+ 1883us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.105523 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.105525 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.105526 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.105604 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.105878 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.105879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.105879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.106104 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:15.106252 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.106766 (+ 514us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.106886 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260319 16:34:15.123567 3858 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:34:15.124068 3858 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:34:15.132431 3858 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d: Generated new TSK 0
I20260319 16:34:15.132668 3858 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:34:15.189069 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39411
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:34:15.263965 3876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:15.264158 3876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:15.264182 3876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:15.265815 3876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:15.265859 3876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:15.265885 3876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:34:15.267554 3876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:15.267769 3876 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:15.267940 3876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:15.270303 3882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:15.270334 3876 server_base.cc:1060] running on GCE node
W20260319 16:34:15.270303 3884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:15.270366 3881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:15.270715 3876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:15.270910 3876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:15.272073 3876 hybrid_clock.cc:648] HybridClock initialized: now 1773938055272050 us; error 31 us; skew 500 ppm
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938055, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:34:15.274426 3876 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:34:15.274928 3876 webserver.cc:492] Webserver started at http://127.24.187.193:45665/ using document root <none> and password file <none>
I20260319 16:34:15.275091 3876 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:15.275125 3876 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:15.275223 3876 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:15.276106 3876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "dff3ee88b5d7491db8c9f3a38a0f057c"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "5fe36b3bf99cc58308080d68782ebf34"
server_key_iv: "fbb3d711e05699f2c9e1bfc4250cc163"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.276403 3876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "dff3ee88b5d7491db8c9f3a38a0f057c"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "5fe36b3bf99cc58308080d68782ebf34"
server_key_iv: "fbb3d711e05699f2c9e1bfc4250cc163"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.277570 3876 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:34:15.278355 3891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.278522 3876 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:34:15.278590 3876 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "dff3ee88b5d7491db8c9f3a38a0f057c"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "5fe36b3bf99cc58308080d68782ebf34"
server_key_iv: "fbb3d711e05699f2c9e1bfc4250cc163"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.278648 3876 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:15.292683 3876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:15.294411 3876 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:15.294530 3876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:15.294734 3876 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:15.295010 3876 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:15.295040 3876 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.295068 3876 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:15.295092 3876 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.300477 3876 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:38729
I20260319 16:34:15.300566 4004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:38729 every 8 connection(s)
I20260319 16:34:15.300815 3876 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:34:15.303414 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 3876
I20260319 16:34:15.303516 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:34:15.303782 25327 external_mini_cluster.cc:1468] Setting key 75c94111d3b6efa9222227425204951e
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938055, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:34:15.310516 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.301610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:53845 (local address 127.24.187.254:39411)
0319 16:34:15.301695 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.301698 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.301797 (+ 99us) server_negotiation.cc:408] Connection header received
0319 16:34:15.302326 (+ 529us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.302328 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.302338 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.302402 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:15.303259 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.304146 (+ 887us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.304876 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.305024 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.307785 (+ 2761us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:15.307790 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:15.307790 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:34:15.307799 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:15.309351 (+ 1552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.309732 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.309733 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.309734 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.309761 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.309936 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.309936 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.309937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.310055 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:15.310121 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.310324 (+ 203us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.310421 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260319 16:34:15.310586 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.301522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39411 (local address 127.24.187.193:53845)
0319 16:34:15.301667 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.301702 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.302207 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.302523 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.302527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.302834 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:15.303113 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.303117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.304249 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.304250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.304779 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.304782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.304917 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.305405 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:15.305429 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:15.307687 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:15.309448 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:34:15.309453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:34:15.309455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:34:15.309648 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:34:15.309820 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:34:15.309821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:34:15.309821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:34:15.309877 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:34:15.310108 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:15.310110 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:15.310213 (+ 103us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.310292 (+ 79us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.310426 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260319 16:34:15.311070 4005 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39411
I20260319 16:34:15.311160 4005 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:15.311343 4005 heartbeater.cc:507] Master 127.24.187.254:39411 requested a full tablet report, sending...
I20260319 16:34:15.311861 3812 ts_manager.cc:194] Registered new tserver with Master: dff3ee88b5d7491db8c9f3a38a0f057c (127.24.187.193:38729)
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
I20260319 16:34:15.312747 3812 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:53845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:34:15.353026 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39411
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:34:15.426872 4012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:15.427045 4012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:15.427074 4012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:15.428536 4012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:15.428587 4012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:15.428622 4012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:34:15.430229 4012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:15.430436 4012 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:15.430598 4012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:34:15.432981 4020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:15.433127 4012 server_base.cc:1060] running on GCE node
W20260319 16:34:15.432984 4017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:15.432979 4018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:15.433413 4012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:15.433621 4012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:15.434751 4012 hybrid_clock.cc:648] HybridClock initialized: now 1773938055434735 us; error 33 us; skew 500 ppm
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938055, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:34:15.436998 4012 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:34:15.437449 4012 webserver.cc:492] Webserver started at http://127.24.187.194:33897/ using document root <none> and password file <none>
I20260319 16:34:15.437604 4012 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:15.437634 4012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:15.437722 4012 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:15.438596 4012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "ed6c256bdd4446c79b26bff22b500ce2"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "b094178d9c6edf9adae332601a53f6eb"
server_key_iv: "55ab337bd52ba56d4c5a27dd08cabb62"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.438851 4012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "ed6c256bdd4446c79b26bff22b500ce2"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "b094178d9c6edf9adae332601a53f6eb"
server_key_iv: "55ab337bd52ba56d4c5a27dd08cabb62"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.439927 4012 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:34:15.440686 4027 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.440851 4012 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:34:15.440920 4012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "ed6c256bdd4446c79b26bff22b500ce2"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "b094178d9c6edf9adae332601a53f6eb"
server_key_iv: "55ab337bd52ba56d4c5a27dd08cabb62"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.440976 4012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:15.459385 4012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:15.461130 4012 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:15.461230 4012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:15.461398 4012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:15.461650 4012 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:15.461676 4012 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.461696 4012 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:15.461735 4012 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.466943 4012 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:36069
I20260319 16:34:15.467036 4141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:36069 every 8 connection(s)
I20260319 16:34:15.467260 4012 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:34:15.467504 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 4012
I20260319 16:34:15.467612 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:34:15.467809 25327 external_mini_cluster.cc:1468] Setting key 9abe3da7b644f5b0f0c9184a3079dcc1
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938055, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:34:15.476011 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.467944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:57059 (local address 127.24.187.254:39411)
0319 16:34:15.467994 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.467996 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.468180 (+ 184us) server_negotiation.cc:408] Connection header received
0319 16:34:15.468746 (+ 566us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.468747 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.468756 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.468792 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:15.469743 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.470225 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.470922 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.471044 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.473437 (+ 2393us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:15.473442 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:15.473443 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:34:15.473450 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:15.474750 (+ 1300us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.475194 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.475195 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.475196 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.475226 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.475431 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.475431 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.475432 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.475527 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:15.475577 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.475868 (+ 291us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.475946 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
I20260319 16:34:15.476255 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.467864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39411 (local address 127.24.187.194:57059)
0319 16:34:15.468068 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.468099 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.468651 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.468846 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.468849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.469245 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:15.469605 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.469609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.470293 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.470294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.470839 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.470841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.470931 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.471374 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:15.471404 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:15.473329 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:15.474831 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:34:15.474836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:34:15.474839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:34:15.475095 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:34:15.475294 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:34:15.475295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:34:15.475295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:34:15.475369 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:34:15.475585 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:15.475590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:15.475741 (+ 151us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.475858 (+ 117us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.475984 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260319 16:34:15.476675 4142 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39411
I20260319 16:34:15.476759 4142 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:15.476910 4142 heartbeater.cc:507] Master 127.24.187.254:39411 requested a full tablet report, sending...
I20260319 16:34:15.477346 3812 ts_manager.cc:194] Registered new tserver with Master: ed6c256bdd4446c79b26bff22b500ce2 (127.24.187.194:36069)
I20260319 16:34:15.477742 3812 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:57059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:34:15.514868 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39411
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:34:15.590385 4149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:34:15.590543 4149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:34:15.590564 4149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:34:15.592046 4149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:34:15.592089 4149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:34:15.592113 4149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:34:15.593698 4149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:34:15.593900 4149 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:34:15.594051 4149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:34:15.596614 4149 server_base.cc:1060] running on GCE node
W20260319 16:34:15.596556 4157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:15.596742 4154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:34:15.596628 4155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:34:15.596975 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:34:15.597158 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:34:15.598294 4149 hybrid_clock.cc:648] HybridClock initialized: now 1773938055598286 us; error 31 us; skew 500 ppm
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938055, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:34:15.600611 4149 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:34:15.601083 4149 webserver.cc:492] Webserver started at http://127.24.187.195:46587/ using document root <none> and password file <none>
I20260319 16:34:15.601243 4149 fs_manager.cc:362] Metadata directory not provided
I20260319 16:34:15.601277 4149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:34:15.601374 4149 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:34:15.602288 4149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "4a0b431ac46647d69d2b90bf7a823d50"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "2532dc6a67fbb54aa3d665ed5f1078b2"
server_key_iv: "c11a53c75891b2ec82e580493e0e14c4"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.602595 4149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "4a0b431ac46647d69d2b90bf7a823d50"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "2532dc6a67fbb54aa3d665ed5f1078b2"
server_key_iv: "c11a53c75891b2ec82e580493e0e14c4"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.603770 4149 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:34:15.604441 4164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.604590 4149 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:34:15.604648 4149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "4a0b431ac46647d69d2b90bf7a823d50"
format_stamp: "Formatted at 2026-03-19 16:34:15 on dist-test-slave-svn8"
server_key: "2532dc6a67fbb54aa3d665ed5f1078b2"
server_key_iv: "c11a53c75891b2ec82e580493e0e14c4"
server_key_version: "encryptionkey@0"
I20260319 16:34:15.604708 4149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:34:15.625756 4149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:34:15.627779 4149 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:34:15.627883 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:34:15.628055 4149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:34:15.628307 4149 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:34:15.628335 4149 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.628356 4149 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:34:15.628371 4149 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:34:15.633778 4149 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:39765
I20260319 16:34:15.633821 4277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:39765 every 8 connection(s)
I20260319 16:34:15.634091 4149 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[3764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938055, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:34:15.639444 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 4149
I20260319 16:34:15.639528 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:34:15.639705 25327 external_mini_cluster.cc:1468] Setting key 0f18f6404dd19f6089fc4fc7753a5298
I20260319 16:34:15.642473 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.634759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:43533 (local address 127.24.187.254:39411)
0319 16:34:15.634811 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.634813 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.634909 (+ 96us) server_negotiation.cc:408] Connection header received
0319 16:34:15.635373 (+ 464us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.635374 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.635385 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.635446 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:15.636161 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.636678 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.637257 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.637391 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.639854 (+ 2463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:15.639859 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:15.639859 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:34:15.639868 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:15.641130 (+ 1262us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.641556 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.641557 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.641558 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.641586 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.641878 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.641879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.641879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.641975 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:15.642041 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.642296 (+ 255us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.642411 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:34:15.642591 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.634677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39411 (local address 127.24.187.195:43533)
0319 16:34:15.634814 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.634844 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.635298 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.635497 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.635500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.635802 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:34:15.636069 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.636072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.636785 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.636787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.637168 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.637169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.637266 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.637761 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:34:15.637778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:34:15.639722 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:34:15.641214 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:34:15.641218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:34:15.641220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:34:15.641452 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:34:15.641691 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:34:15.641692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:34:15.641693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:34:15.641805 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:34:15.642057 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:34:15.642066 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:34:15.642174 (+ 108us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.642269 (+ 95us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.642412 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260319 16:34:15.643050 4278 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39411
I20260319 16:34:15.643137 4278 heartbeater.cc:461] Registering TS with master...
I20260319 16:34:15.643313 4278 heartbeater.cc:507] Master 127.24.187.254:39411 requested a full tablet report, sending...
I20260319 16:34:15.643721 3812 ts_manager.cc:194] Registered new tserver with Master: 4a0b431ac46647d69d2b90bf7a823d50 (127.24.187.195:39765)
I20260319 16:34:15.644069 3812 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:43533
I20260319 16:34:15.651552 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:34:15.657749 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.652793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.24.187.254:39411)
0319 16:34:15.652840 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.652842 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.652886 (+ 44us) server_negotiation.cc:408] Connection header received
0319 16:34:15.652958 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.652960 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.652969 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.653011 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:34:15.653672 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.654172 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.654702 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.654859 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.655717 (+ 858us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:34:15.655720 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:34:15.655721 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:34:15.655727 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:34:15.656818 (+ 1091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.657102 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.657103 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.657104 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.657126 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:34:15.657254 (+ 128us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:34:15.657254 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:34:15.657255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:34:15.657365 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:34:15.657460 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.657542 (+ 82us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.657617 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:34:15.659209 3812 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:34:15.660455 3812 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:34:15.665822 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.662726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:38729 (local address 127.0.0.1:33064)
0319 16:34:15.662918 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.662950 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.663019 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.663228 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.663230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.663233 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:34:15.663494 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.663496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.664542 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.664543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.665493 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.665496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.665571 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.665625 (+ 54us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.665670 (+ 45us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.665738 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":58,"threads_started":1}
I20260319 16:34:15.666528 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.662886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.24.187.194:36069)
0319 16:34:15.663046 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.663054 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.663155 (+ 101us) server_negotiation.cc:408] Connection header received
0319 16:34:15.663197 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.663199 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.663221 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.663272 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:34:15.663579 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.664646 (+ 1067us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.665793 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.666320 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.666381 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.666412 (+ 31us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.666453 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260319 16:34:15.666563 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.662882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.24.187.193:38729)
0319 16:34:15.663054 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.663061 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.663071 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:34:15.663110 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.663112 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.663129 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.663205 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:34:15.663585 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.664464 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.666014 (+ 1550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.666409 (+ 395us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.666442 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.666468 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.666507 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260319 16:34:15.665830 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.662726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:36069 (local address 127.0.0.1:47166)
0319 16:34:15.663088 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.663096 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.663147 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.663281 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.663281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.663283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:34:15.663496 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.663498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.664796 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.664797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.665596 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.665598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.665678 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.665681 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.665748 (+ 67us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.665781 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260319 16:34:15.666808 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.662974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.24.187.195:39765)
0319 16:34:15.663130 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.663136 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.663430 (+ 294us) server_negotiation.cc:408] Connection header received
0319 16:34:15.663561 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.663564 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.663582 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.663642 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:34:15.663994 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.664984 (+ 990us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.666078 (+ 1094us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.666559 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.666657 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.666682 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.666726 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260319 16:34:15.665992 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.662749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:39765 (local address 127.0.0.1:37168)
0319 16:34:15.663349 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.663358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.663416 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.663730 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.663731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.663733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:34:15.663919 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.663920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.665056 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.665058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.665797 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.665800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.665858 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.665862 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.665889 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.665924 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":106,"threads_started":1}
I20260319 16:34:15.667250 4076 tablet_service.cc:1511] Processing CreateTablet for tablet 04b090b1add94d26a74382e6d2f3e397 (DEFAULT_TABLE table=test-table [id=146ce4ef723b4729a4a3743e892e2b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:34:15.667254 3939 tablet_service.cc:1511] Processing CreateTablet for tablet 04b090b1add94d26a74382e6d2f3e397 (DEFAULT_TABLE table=test-table [id=146ce4ef723b4729a4a3743e892e2b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:34:15.667507 3939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b090b1add94d26a74382e6d2f3e397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:34:15.667511 4076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b090b1add94d26a74382e6d2f3e397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:34:15.667569 4212 tablet_service.cc:1511] Processing CreateTablet for tablet 04b090b1add94d26a74382e6d2f3e397 (DEFAULT_TABLE table=test-table [id=146ce4ef723b4729a4a3743e892e2b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:34:15.667774 4212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04b090b1add94d26a74382e6d2f3e397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:34:15.669611 4296 tablet_bootstrap.cc:492] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c: Bootstrap starting.
I20260319 16:34:15.670167 4296 tablet_bootstrap.cc:654] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c: Neither blocks nor log segments found. Creating new log.
I20260319 16:34:15.670154 4297 tablet_bootstrap.cc:492] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: Bootstrap starting.
I20260319 16:34:15.670445 4296 log.cc:826] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c: Log is configured to *not* fsync() on all Append() calls
I20260319 16:34:15.670681 4297 tablet_bootstrap.cc:654] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: Neither blocks nor log segments found. Creating new log.
I20260319 16:34:15.670933 4297 log.cc:826] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: Log is configured to *not* fsync() on all Append() calls
I20260319 16:34:15.671226 4296 tablet_bootstrap.cc:492] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c: No bootstrap required, opened a new log
I20260319 16:34:15.671211 4298 tablet_bootstrap.cc:492] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Bootstrap starting.
I20260319 16:34:15.671286 4296 ts_tablet_manager.cc:1403] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:34:15.671744 4298 tablet_bootstrap.cc:654] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Neither blocks nor log segments found. Creating new log.
I20260319 16:34:15.672022 4298 log.cc:826] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Log is configured to *not* fsync() on all Append() calls
I20260319 16:34:15.672483 4297 tablet_bootstrap.cc:492] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: No bootstrap required, opened a new log
I20260319 16:34:15.672547 4297 ts_tablet_manager.cc:1403] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:34:15.672797 4296 raft_consensus.cc:359] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.672937 4296 raft_consensus.cc:385] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:34:15.672966 4296 raft_consensus.cc:740] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff3ee88b5d7491db8c9f3a38a0f057c, State: Initialized, Role: FOLLOWER
I20260319 16:34:15.673044 4296 consensus_queue.cc:260] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.673520 4296 ts_tablet_manager.cc:1434] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:34:15.673565 4005 heartbeater.cc:499] Master 127.24.187.254:39411 was elected leader, sending a full tablet report...
I20260319 16:34:15.674016 4298 tablet_bootstrap.cc:492] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: No bootstrap required, opened a new log
I20260319 16:34:15.674081 4298 ts_tablet_manager.cc:1403] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260319 16:34:15.674280 4297 raft_consensus.cc:359] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.674448 4297 raft_consensus.cc:385] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:34:15.674491 4297 raft_consensus.cc:740] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6c256bdd4446c79b26bff22b500ce2, State: Initialized, Role: FOLLOWER
I20260319 16:34:15.674572 4297 consensus_queue.cc:260] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.674782 4297 ts_tablet_manager.cc:1434] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:34:15.674813 4142 heartbeater.cc:499] Master 127.24.187.254:39411 was elected leader, sending a full tablet report...
I20260319 16:34:15.675421 4298 raft_consensus.cc:359] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.675529 4298 raft_consensus.cc:385] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:34:15.675547 4298 raft_consensus.cc:740] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0b431ac46647d69d2b90bf7a823d50, State: Initialized, Role: FOLLOWER
I20260319 16:34:15.675648 4298 consensus_queue.cc:260] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.675858 4298 ts_tablet_manager.cc:1434] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:34:15.675885 4278 heartbeater.cc:499] Master 127.24.187.254:39411 was elected leader, sending a full tablet report...
W20260319 16:34:15.717867 4143 tablet.cc:2379] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:34:15.735978 4302 raft_consensus.cc:493] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:34:15.736137 4302 raft_consensus.cc:515] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.736558 4302 leader_election.cc:290] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed6c256bdd4446c79b26bff22b500ce2 (127.24.187.194:36069), 4a0b431ac46647d69d2b90bf7a823d50 (127.24.187.195:39765)
I20260319 16:34:15.739365 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.736760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:39765 (local address 127.24.187.193:37671)
0319 16:34:15.736810 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.736823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.736865 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.737049 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.737050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.737052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:34:15.737278 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.737280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.738172 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.738174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.739086 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.739090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.739164 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.739170 (+ 6us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.739199 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.739237 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:34:15.739395 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.736762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:36069 (local address 127.24.187.193:46269)
0319 16:34:15.736990 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.736996 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0319 16:34:15.737048 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:34:15.737157 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:34:15.737157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:34:15.737158 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:34:15.737436 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.737437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.738320 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.738321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:34:15.739127 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:34:15.739130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.739200 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.739204 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:34:15.739233 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:34:15.739269 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":52,"thread_start_us":118,"threads_started":1}
I20260319 16:34:15.739940 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.736844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:37671 (local address 127.24.187.195:39765)
0319 16:34:15.736906 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.736908 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.736916 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:34:15.736943 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.736944 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.736953 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.737027 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:34:15.737361 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.738083 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.739209 (+ 1126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.739811 (+ 602us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.739821 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.739846 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.739877 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:34:15.740108 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.736805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:46269 (local address 127.24.187.194:36069)
0319 16:34:15.736849 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.736851 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.737059 (+ 208us) server_negotiation.cc:408] Connection header received
0319 16:34:15.737090 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.737091 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.737101 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.737140 (+ 39us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:34:15.737502 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.738215 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.739548 (+ 1333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.739991 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.739999 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.740020 (+ 21us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.740051 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:34:15.740350 4232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04b090b1add94d26a74382e6d2f3e397" candidate_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0b431ac46647d69d2b90bf7a823d50" is_pre_election: true
I20260319 16:34:15.740527 4232 raft_consensus.cc:2468] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff3ee88b5d7491db8c9f3a38a0f057c in term 0.
I20260319 16:34:15.740625 4096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04b090b1add94d26a74382e6d2f3e397" candidate_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6c256bdd4446c79b26bff22b500ce2" is_pre_election: true
I20260319 16:34:15.740779 4096 raft_consensus.cc:2468] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff3ee88b5d7491db8c9f3a38a0f057c in term 0.
I20260319 16:34:15.740762 3894 leader_election.cc:304] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a0b431ac46647d69d2b90bf7a823d50, dff3ee88b5d7491db8c9f3a38a0f057c; no voters:
I20260319 16:34:15.740888 4302 raft_consensus.cc:2804] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:34:15.741041 4302 raft_consensus.cc:493] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:34:15.741068 4302 raft_consensus.cc:3060] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:34:15.742054 4302 raft_consensus.cc:515] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.742187 4302 leader_election.cc:290] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [CANDIDATE]: Term 1 election: Requested vote from peers ed6c256bdd4446c79b26bff22b500ce2 (127.24.187.194:36069), 4a0b431ac46647d69d2b90bf7a823d50 (127.24.187.195:39765)
I20260319 16:34:15.742357 4096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04b090b1add94d26a74382e6d2f3e397" candidate_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6c256bdd4446c79b26bff22b500ce2"
I20260319 16:34:15.742385 4232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04b090b1add94d26a74382e6d2f3e397" candidate_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a0b431ac46647d69d2b90bf7a823d50"
I20260319 16:34:15.742437 4096 raft_consensus.cc:3060] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:34:15.742455 4232 raft_consensus.cc:3060] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:34:15.743079 4096 raft_consensus.cc:2468] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff3ee88b5d7491db8c9f3a38a0f057c in term 1.
I20260319 16:34:15.743119 4232 raft_consensus.cc:2468] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff3ee88b5d7491db8c9f3a38a0f057c in term 1.
I20260319 16:34:15.743240 3892 leader_election.cc:304] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff3ee88b5d7491db8c9f3a38a0f057c, ed6c256bdd4446c79b26bff22b500ce2; no voters:
I20260319 16:34:15.743335 4302 raft_consensus.cc:2804] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:34:15.743498 4302 raft_consensus.cc:697] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [term 1 LEADER]: Becoming Leader. State: Replica: dff3ee88b5d7491db8c9f3a38a0f057c, State: Running, Role: LEADER
I20260319 16:34:15.743575 4302 consensus_queue.cc:237] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } }
I20260319 16:34:15.744374 3812 catalog_manager.cc:5671] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c reported cstate change: term changed from 0 to 1, leader changed from <none> to dff3ee88b5d7491db8c9f3a38a0f057c (127.24.187.193). New cstate: current_term: 1 leader_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff3ee88b5d7491db8c9f3a38a0f057c" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 38729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 } health_report { overall_health: UNKNOWN } } }
I20260319 16:34:15.758807 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.756823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.24.187.193:38729)
0319 16:34:15.756871 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:34:15.756873 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:34:15.756879 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:34:15.756919 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:34:15.756920 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:34:15.756935 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:34:15.756988 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:34:15.757254 (+ 266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.757792 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:34:15.758353 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:34:15.758493 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:34:15.758531 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:34:15.758616 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:34:15.758678 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0319 16:34:15.758727 (+ 49us) server_negotiation.cc:300] Negotiation successful
0319 16:34:15.758749 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:34:15.760387 4232 raft_consensus.cc:1275] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 1 FOLLOWER]: Refusing update from remote peer dff3ee88b5d7491db8c9f3a38a0f057c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:34:15.760411 4096 raft_consensus.cc:1275] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 1 FOLLOWER]: Refusing update from remote peer dff3ee88b5d7491db8c9f3a38a0f057c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:34:15.760691 4306 consensus_queue.cc:1048] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6c256bdd4446c79b26bff22b500ce2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:34:15.760805 4302 consensus_queue.cc:1048] T 04b090b1add94d26a74382e6d2f3e397 P dff3ee88b5d7491db8c9f3a38a0f057c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a0b431ac46647d69d2b90bf7a823d50" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:34:15.762894 4309 mvcc.cc:204] Tried to move back new op lower bound from 7266050276392026112 to 7266050276326293504. Current Snapshot: MvccSnapshot[applied={T|T < 7266050276392026112}]
I20260319 16:34:15.763371 4310 mvcc.cc:204] Tried to move back new op lower bound from 7266050276392026112 to 7266050276326293504. Current Snapshot: MvccSnapshot[applied={T|T < 7266050276392026112 or (T in {7266050276392026112})}]
I20260319 16:34:15.765377 3812 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:34:15.765468 3812 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:34:15.766294 3812 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 27fa7e491bfe4513877ea02912cfd70d: Sending DeleteTablet for 3 replicas of tablet 04b090b1add94d26a74382e6d2f3e397
I20260319 16:34:15.766639 4076 tablet_service.cc:1558] Processing DeleteTablet for tablet 04b090b1add94d26a74382e6d2f3e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:34:15 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:47166
I20260319 16:34:15.766683 3939 tablet_service.cc:1558] Processing DeleteTablet for tablet 04b090b1add94d26a74382e6d2f3e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:34:15 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33064
I20260319 16:34:15.766731 4212 tablet_service.cc:1558] Processing DeleteTablet for tablet 04b090b1add94d26a74382e6d2f3e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:34:15 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:37168
I20260319 16:34:15.767073 4321 tablet_replica.cc:333] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: stopping tablet replica
I20260319 16:34:15.767125 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 3876
I20260319 16:34:15.767158 4321 raft_consensus.cc:2243] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:34:15.767247 4321 raft_consensus.cc:2272] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:34:15.768563 4321 ts_tablet_manager.cc:1916] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:34:15.770174 4321 ts_tablet_manager.cc:1929] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:34:15.770255 4321 log.cc:1199] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/04b090b1add94d26a74382e6d2f3e397
I20260319 16:34:15.770534 4321 ts_tablet_manager.cc:1950] T 04b090b1add94d26a74382e6d2f3e397 P 4a0b431ac46647d69d2b90bf7a823d50: Deleting consensus metadata
I20260319 16:34:15.771132 3798 catalog_manager.cc:5002] TS 4a0b431ac46647d69d2b90bf7a823d50 (127.24.187.195:39765): tablet 04b090b1add94d26a74382e6d2f3e397 (table test-table [id=146ce4ef723b4729a4a3743e892e2b56]) successfully deleted
W20260319 16:34:15.773331 3797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:38729 (error 108)
I20260319 16:34:15.773380 4319 tablet_replica.cc:333] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2: stopping tablet replica
I20260319 16:34:15.773480 4319 raft_consensus.cc:2243] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:34:15.773587 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 4012
I20260319 16:34:15.773609 4319 raft_consensus.cc:2272] T 04b090b1add94d26a74382e6d2f3e397 P ed6c256bdd4446c79b26bff22b500ce2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:34:15.774565 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.774414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:38729 (local address 127.0.0.1:33082)
0319 16:34:15.774483 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.774528 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260319 16:34:15.774711 3797 catalog_manager.cc:4729] TS dff3ee88b5d7491db8c9f3a38a0f057c (127.24.187.193:38729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04b090b1add94d26a74382e6d2f3e397: Network error: Client connection negotiation failed: client connection to 127.24.187.193:38729: connect: Connection refused (error 111)
I20260319 16:34:15.778424 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 4149
I20260319 16:34:15.779289 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:34:15.778563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:36069 (local address 127.0.0.1:47180)
0319 16:34:15.779223 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0319 16:34:15.779251 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.194:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640}
I20260319 16:34:15.783205 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 3780
2026-03-19T16:34:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2781 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[4325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[4325](info): set up 2 sockets
Mar 19 16:34:15 dist-test-slave-svn8 krb5kdc[4325](info): commencing operation
krb5kdc: starting...
W20260319 16:34:17.618850 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:34:17 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:34:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:34:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/rangeradmin_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/rangeradmin_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/rangerlookup_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/rangerlookup_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
I20260319 16:34:17.781570 25327 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskq2KZu_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/postgres -l logfile start
2026-03-19 16:34:20.601 UTC [4356] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-03-19 16:34:20.601 UTC [4356] LOG: listening on IPv4 address "127.24.187.212", port 48225
2026-03-19 16:34:20.605 UTC [4356] LOG: listening on Unix socket "/tmp/.s.PGSQL.48225"
2026-03-19 16:34:20.612 UTC [4361] LOG: database system was shut down at 2026-03-19 16:34:18 UTC
2026-03-19 16:34:20.618 UTC [4356] LOG: database system is ready to accept connections
I20260319 16:34:22.374728 25327 mini_postgres.cc:96] Postgres bound to 48225
2026-03-19 16:34:22.380 UTC [4367] FATAL: database "slave" does not exist
127.24.187.212:48225 - accepting connections
I20260319 16:34:22.381475 25327 mini_ranger.cc:161] Starting Ranger...
I20260319 16:34:22.400779 25327 mini_ranger.cc:84] Created miniranger Postgres user
I20260319 16:34:22.473810 25327 mini_ranger.cc:87] Created ranger Postgres database
I20260319 16:34:22.473898 25327 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-admin
2026-03-19 16:34:22,748 [I] DB FLAVOR :POSTGRES
2026-03-19 16:34:22,749 [I] --------- Verifying Ranger DB connection ---------
2026-03-19 16:34:22,749 [I] Checking connection..
2026-03-19 16:34:22,749 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-03-19 16:34:23,097 [I] Checking connection passed.
2026-03-19 16:34:23,097 [I] --------- Verifying version history table ---------
2026-03-19 16:34:23,098 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-19 16:34:23,420 [I] Table x_db_version_h does not exist in database ranger
2026-03-19 16:34:23,421 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-19 16:34:23,738 [I] Table x_db_version_h does not exist in database ranger
2026-03-19 16:34:23,738 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-03-19 16:34:23,738 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-03-19 16:34:24.046 UTC [4480] WARNING: there is no transaction in progress
2026-03-19 16:34:24,059 [I] create_dbversion_catalog.sql file imported successfully
2026-03-19 16:34:24,059 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-19 16:34:24,409 [I] Table x_db_version_h already exists in database 'ranger'
2026-03-19 16:34:24,410 [I] --------- Importing Ranger Core DB Schema ---------
2026-03-19 16:34:24,410 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-03-19 16:34:24,741 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-03-19 16:34:25,072 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-svn8.c.gcp-upstream.internal','N') ;"
2026-03-19 16:34:25,361 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-03-19 16:34:25,667 [I] Table x_portal_user does not exist in database ranger
2026-03-19 16:34:25,668 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-03-19 16:34:26,008 [I] Table x_policy_ref_group does not exist in database ranger
2026-03-19 16:34:26,008 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-03-19 16:34:26,364 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-03-19 16:34:26,694 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-03-19 16:34:26,695 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-03-19 16:34:27.585 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.596 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.609 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.621 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.637 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.652 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.702 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.708 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.719 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.729 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.738 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.749 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.758 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.763 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.767 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:27.772 UTC [4680] WARNING: there is no transaction in progress
2026-03-19 16:34:28,237 [I] ranger_core_db_postgres.sql file imported successfully
2026-03-19 16:34:28,237 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-03-19 16:34:28,532 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-03-19 16:34:28,533 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-03-19 16:34:28,866 [I] Table x_portal_user already exists in database 'ranger'
2026-03-19 16:34:28,867 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-03-19 16:34:29,181 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-03-19 16:34:29,181 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-03-19 16:34:29,489 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-03-19 16:34:29,797 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-svn8.c.gcp-upstream.internal';"
2026-03-19 16:34:30,114 [I] CORE_DB_SCHEMA import status has been updated
2026-03-19 16:34:30,115 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-03-19 16:34:30,453 [I] DB_PATCHES have already been applied
I20260319 16:34:30.458645 25327 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-admin:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskq2KZu_/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260319 16:34:30.458710 25327 mini_ranger.cc:193] Using host: 127.24.187.212
I20260319 16:34:30.461524 25327 mini_ranger.cc:239] Ranger admin URL: http://127.24.187.212:45689
Mar 19, 2026 4:34:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Mar 19, 2026 4:34:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Mar 19, 2026 4:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp
Mar 19, 2026 4:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Mar 19, 2026 4:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp] .....
Mar 19, 2026 4:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp].
Mar 19, 2026 4:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260319 16:34:32.513425 25327 mini_ranger.cc:160] Time spent starting Ranger: real 10.132s user 0.000s sys 0.010s
Mar 19, 2026 4:34:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Mar 19, 2026 4:34:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XUserREST
Mar 19, 2026 4:34:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Mar 19, 2026 4:34:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Mar 19, 2026 4:34:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260319 16:34:56.961620 25327 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/rangerkms_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/rangerkms_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.187.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.187.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Mar 19 16:34:57 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938097, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260319 16:34:57.044920 25327 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260319 16:34:57.054755 25327 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260319 16:34:57.160347 25327 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260319 16:34:57.160430 25327 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms
I20260319 16:34:57.160458 25327 mini_ranger_kms.cc:227] Using postgres at 127.24.187.212:48225
2026-03-19 16:34:57,885 [I] DB FLAVOR :POSTGRES
2026-03-19 16:34:57,885 [I] --------- Verifying Ranger DB connection ---------
2026-03-19 16:34:57,885 [I] Checking connection
2026-03-19 16:34:57,885 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-03-19 16:34:58,210 [I] connection success
2026-03-19 16:34:58,210 [I] --------- Verifying Ranger DB tables ---------
2026-03-19 16:34:58,210 [I] Verifying table ranger_masterkey in database rangerkms
2026-03-19 16:34:58,210 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-03-19 16:34:58,563 [I] Table ranger_masterkey does not exist in database rangerkms
2026-03-19 16:34:58,563 [I] --------- Importing Ranger Core DB Schema ---------
2026-03-19 16:34:58,563 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-03-19 16:34:58,563 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:48225/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-03-19 16:34:58,885 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260319 16:34:59.423064 25327 mini_ranger_kms.cc:326] Created kms service
I20260319 16:34:59.645360 25327 mini_ranger_kms.cc:342] Created kudu user
I20260319 16:34:59.698985 25327 mini_ranger_kms.cc:359] Created rangerkms user
Mar 19 16:34:59 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938097, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:35:00.193742 25327 mini_ranger_kms.cc:400] Added ranger policy
I20260319 16:35:00.193892 25327 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms:/tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260319 16:35:00.193909 25327 mini_ranger_kms.cc:240] Using host: 127.24.187.212
I20260319 16:35:00.197647 25327 mini_ranger_kms.cc:292] Ranger KMS PID: 4982
I20260319 16:35:00.197764 25327 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.24.187.212:46889
Mar 19, 2026 4:35:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Mar 19, 2026 4:35:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Mar 19, 2026 4:35:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Mar 19, 2026 4:35:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms/ews/webapp] .....
Mar 19, 2026 4:35:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-kms/ews/webapp].
Mar 19, 2026 4:35:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260319 16:35:02.312461 25327 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.268s user 0.013s sys 0.229s
I20260319 16:35:02.312561 25327 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260319 16:35:02.312606 25327 mini_ranger_kms.cc:417] 127.24.187.212:46889/kms/v1/keys
Mar 19 16:35:02 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938102, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.187.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Mar 19 16:35:03 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938102, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.187.212@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
Mar 19 16:35:04 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938102, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.187.212@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
Mar 19 16:35:06 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:35:06.744931 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:43137
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:43137
--ranger_config_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:46889/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:35:06.844813 5051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:35:06.845001 5051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:35:06.845022 5051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:35:06.846596 5051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:35:06.846644 5051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:35:06.846665 5051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:35:06.846676 5051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:35:06.846688 5051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:35:06.848430 5051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.187.212:46889/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:43137
--ranger_config_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:43137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:35:06.848624 5051 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:35:06.848809 5051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:35:06.851409 5057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:35:06.851522 5059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:35:06.851662 5051 server_base.cc:1060] running on GCE node
W20260319 16:35:06.851789 5056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:35:06.852103 5051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:35:06.852375 5051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:35:06.853524 5051 hybrid_clock.cc:648] HybridClock initialized: now 1773938106853505 us; error 33 us; skew 500 ppm
Mar 19 16:35:06 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938106, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:35:06.856211 5051 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:35:06.856775 5051 webserver.cc:492] Webserver started at http://127.24.187.254:41261/ using document root <none> and password file <none>
I20260319 16:35:06.856947 5051 fs_manager.cc:362] Metadata directory not provided
I20260319 16:35:06.857019 5051 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:35:06.857156 5051 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:35:06 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938106, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:35:06.918308 5051 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "569e3ea5175b43c7bf27bf774009a622"
format_stamp: "Formatted at 2026-03-19 16:35:06 on dist-test-slave-svn8"
server_key: "4f091f53b99fafde18c66fa839f51555"
server_key_iv: "dd602b5394a470c7f4ce123d6fbf1f45"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:06.918726 5051 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "569e3ea5175b43c7bf27bf774009a622"
format_stamp: "Formatted at 2026-03-19 16:35:06 on dist-test-slave-svn8"
server_key: "4f091f53b99fafde18c66fa839f51555"
server_key_iv: "dd602b5394a470c7f4ce123d6fbf1f45"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:06.920308 5051 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:35:06.937354 5066 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:06.937551 5051 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:35:06.937630 5051 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "569e3ea5175b43c7bf27bf774009a622"
format_stamp: "Formatted at 2026-03-19 16:35:06 on dist-test-slave-svn8"
server_key: "4f091f53b99fafde18c66fa839f51555"
server_key_iv: "dd602b5394a470c7f4ce123d6fbf1f45"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:06.937750 5051 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:35:06.972345 5051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:35:06.974663 5051 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:35:06.974802 5051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:35:06.979568 5051 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:43137
I20260319 16:35:06.979810 5118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:43137 every 8 connection(s)
I20260319 16:35:06.980119 5051 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:35:06.980777 5119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:35:06.981917 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 5051
I20260319 16:35:06.982025 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:35:06.982179 25327 external_mini_cluster.cc:1468] Setting key 6523357993b585f432ec458213df3f7f
I20260319 16:35:06.983913 5119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622: Bootstrap starting.
I20260319 16:35:06.984438 5119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622: Neither blocks nor log segments found. Creating new log.
I20260319 16:35:06.984663 5119 log.cc:826] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622: Log is configured to *not* fsync() on all Append() calls
I20260319 16:35:06.985333 5119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622: No bootstrap required, opened a new log
I20260319 16:35:06.986466 5119 raft_consensus.cc:359] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e3ea5175b43c7bf27bf774009a622" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43137 } }
I20260319 16:35:06.986611 5119 raft_consensus.cc:385] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:35:06.986641 5119 raft_consensus.cc:740] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569e3ea5175b43c7bf27bf774009a622, State: Initialized, Role: FOLLOWER
I20260319 16:35:06.986788 5119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e3ea5175b43c7bf27bf774009a622" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43137 } }
I20260319 16:35:06.986855 5119 raft_consensus.cc:399] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:35:06.986904 5119 raft_consensus.cc:493] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:35:06.986949 5119 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:35:06.987955 5119 raft_consensus.cc:515] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e3ea5175b43c7bf27bf774009a622" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43137 } }
I20260319 16:35:06.988070 5119 leader_election.cc:304] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569e3ea5175b43c7bf27bf774009a622; no voters:
I20260319 16:35:06.988227 5119 leader_election.cc:290] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:35:06.988440 5119 sys_catalog.cc:565] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:35:06.990048 5119 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260319 16:35:06.990247 5124 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:35:06.990392 5124 raft_consensus.cc:697] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [term 1 LEADER]: Becoming Leader. State: Replica: 569e3ea5175b43c7bf27bf774009a622, State: Running, Role: LEADER
I20260319 16:35:06.990511 5124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e3ea5175b43c7bf27bf774009a622" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43137 } }
Mar 19 16:35:06 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:35:06.993458 5137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569e3ea5175b43c7bf27bf774009a622. Latest consensus state: current_term: 1 leader_uuid: "569e3ea5175b43c7bf27bf774009a622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e3ea5175b43c7bf27bf774009a622" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43137 } } }
I20260319 16:35:06.993465 5125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569e3ea5175b43c7bf27bf774009a622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e3ea5175b43c7bf27bf774009a622" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43137 } } }
I20260319 16:35:06.994001 5137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [sys.catalog]: This master's current role is: LEADER
I20260319 16:35:06.994027 5125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622 [sys.catalog]: This master's current role is: LEADER
I20260319 16:35:06.994256 5141 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:35:06.994733 5141 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:35:06.997299 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:06.982684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.24.187.254:43137)
0319 16:35:06.982843 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0319 16:35:06.982855 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0319 16:35:06.985010 (+ 2155us) server_negotiation.cc:408] Connection header received
0319 16:35:06.985398 (+ 388us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:35:06.985406 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:35:06.985463 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:35:06.985587 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:35:06.986508 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:35:06.987668 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:35:06.988438 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:35:06.988686 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:35:06.992169 (+ 3483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:35:06.992290 (+ 121us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:35:06.992291 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:35:06.992370 (+ 79us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:35:06.995095 (+ 2725us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:35:06.995483 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:35:06.995486 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:35:06.995487 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:35:06.995537 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:35:06.995727 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:35:06.995729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:35:06.995730 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:35:06.996011 (+ 281us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:35:06.996895 (+ 884us) server_negotiation.cc:1036] Waiting for connection context
0319 16:35:06.996981 (+ 86us) server_negotiation.cc:300] Negotiation successful
0319 16:35:06.997110 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260319 16:35:06.998633 5141 catalog_manager.cc:1357] Generated new cluster ID: af3586fa08164b8780ddf78e6dc3dd63
I20260319 16:35:06.998680 5141 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:35:07.004314 5141 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:35:07.004858 5141 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:35:07.009343 5141 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 569e3ea5175b43c7bf27bf774009a622: Generated new TSK 0
I20260319 16:35:07.009582 5141 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:35:07.944181 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:46889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:43137
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:35:08.026221 5178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:35:08.026458 5178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:35:08.026490 5178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:35:08.028218 5178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:35:08.028282 5178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:35:08.028322 5178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:35:08.030999 5178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.187.212:46889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:35:08.031241 5178 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:35:08.031508 5178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:35:08.034294 5183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:35:08.034309 5184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:35:08.034651 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:35:08.035339 5178 server_base.cc:1060] running on GCE node
I20260319 16:35:08.035521 5178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:35:08.035732 5178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:35:08.036886 5178 hybrid_clock.cc:648] HybridClock initialized: now 1773938108036865 us; error 33 us; skew 500 ppm
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:35:08.039971 5178 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:35:08.040637 5178 webserver.cc:492] Webserver started at http://127.24.187.193:46213/ using document root <none> and password file <none>
I20260319 16:35:08.040848 5178 fs_manager.cc:362] Metadata directory not provided
I20260319 16:35:08.040897 5178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:35:08.040997 5178 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:35:08.058009 5178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "37cb5b42a96b4906b15dd997bb8a08c5"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "cda06a53a2ee0c1917c7473f41b67653"
server_key_iv: "350ab2a6ce224e758e0247e1b5d45154"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.058497 5178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "37cb5b42a96b4906b15dd997bb8a08c5"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "cda06a53a2ee0c1917c7473f41b67653"
server_key_iv: "350ab2a6ce224e758e0247e1b5d45154"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.060122 5178 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:35:08.076294 5193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.076538 5178 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:35:08.076620 5178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "37cb5b42a96b4906b15dd997bb8a08c5"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "cda06a53a2ee0c1917c7473f41b67653"
server_key_iv: "350ab2a6ce224e758e0247e1b5d45154"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.076714 5178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:35:08.100389 5178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:35:08.102219 5178 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:35:08.102399 5178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:35:08.102643 5178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:35:08.102975 5178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:35:08.103017 5178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.103046 5178 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:35:08.103070 5178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.109481 5178 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:43413
I20260319 16:35:08.109544 5306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:43413 every 8 connection(s)
I20260319 16:35:08.109836 5178 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:35:08.114518 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 5178
I20260319 16:35:08.114629 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:35:08.114789 25327 external_mini_cluster.cc:1468] Setting key e78a407988c426333ded6d156b9c5c79
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:35:08.118599 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:08.110545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:36275 (local address 127.24.187.254:43137)
0319 16:35:08.110658 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0319 16:35:08.110660 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:35:08.110731 (+ 71us) server_negotiation.cc:408] Connection header received
0319 16:35:08.111717 (+ 986us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:35:08.111719 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:35:08.111730 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:35:08.111779 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:35:08.112568 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.113096 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:35:08.113661 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.113824 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:35:08.115618 (+ 1794us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:35:08.115622 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:35:08.115623 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:35:08.115650 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:35:08.117161 (+ 1511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:35:08.117656 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:35:08.117657 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:35:08.117658 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:35:08.117682 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:35:08.117926 (+ 244us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:35:08.117927 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:35:08.117927 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:35:08.118044 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:35:08.118111 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:35:08.118434 (+ 323us) server_negotiation.cc:300] Negotiation successful
0319 16:35:08.118529 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":45,"threads_started":1}
I20260319 16:35:08.118688 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:08.110470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:36275)
0319 16:35:08.110605 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:08.110641 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0319 16:35:08.111614 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:35:08.111835 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:35:08.111839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:35:08.112079 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:35:08.112478 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:35:08.112484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.113176 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:35:08.113177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:35:08.113600 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:35:08.113603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.113738 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:35:08.114183 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:35:08.114206 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:35:08.115528 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:35:08.117254 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:35:08.117259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:35:08.117261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:35:08.117577 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:35:08.117781 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:35:08.117782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:35:08.117782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:35:08.117860 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:35:08.118111 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:35:08.118115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:35:08.118258 (+ 143us) client_negotiation.cc:769] Sending connection context
0319 16:35:08.118381 (+ 123us) client_negotiation.cc:240] Negotiation successful
0319 16:35:08.118493 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260319 16:35:08.119192 5307 heartbeater.cc:344] Connected to a master server at 127.24.187.254:43137
I20260319 16:35:08.119284 5307 heartbeater.cc:461] Registering TS with master...
I20260319 16:35:08.119475 5307 heartbeater.cc:507] Master 127.24.187.254:43137 requested a full tablet report, sending...
I20260319 16:35:08.120021 5083 ts_manager.cc:194] Registered new tserver with Master: 37cb5b42a96b4906b15dd997bb8a08c5 (127.24.187.193:43413)
I20260319 16:35:08.120955 5083 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:36275
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:35:08.166699 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:46889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:43137
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:35:08.270713 5315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:35:08.270993 5315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:35:08.271056 5315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:35:08.273303 5315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:35:08.273422 5315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:35:08.273497 5315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:35:08.276007 5315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.187.212:46889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:35:08.276365 5315 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:35:08.276582 5315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:35:08.279667 5322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:35:08.279592 5323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:35:08.279592 5325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:35:08.280643 5315 server_base.cc:1060] running on GCE node
I20260319 16:35:08.280802 5315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:35:08.280997 5315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:35:08.282135 5315 hybrid_clock.cc:648] HybridClock initialized: now 1773938108282106 us; error 38 us; skew 500 ppm
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:35:08.284876 5315 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:35:08.285459 5315 webserver.cc:492] Webserver started at http://127.24.187.194:46223/ using document root <none> and password file <none>
I20260319 16:35:08.285662 5315 fs_manager.cc:362] Metadata directory not provided
I20260319 16:35:08.285698 5315 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:35:08.285830 5315 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:35:08.300473 5315 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "6cc451d324234a6c8537c65242bee2ba"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "e826436e20a23c0d9fe23c108c3f0dc9"
server_key_iv: "02221a6480a8ffa5f470b4905633821d"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.300877 5315 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "6cc451d324234a6c8537c65242bee2ba"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "e826436e20a23c0d9fe23c108c3f0dc9"
server_key_iv: "02221a6480a8ffa5f470b4905633821d"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.302524 5315 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:35:08.333647 5334 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.333958 5315 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20260319 16:35:08.334031 5315 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "6cc451d324234a6c8537c65242bee2ba"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "e826436e20a23c0d9fe23c108c3f0dc9"
server_key_iv: "02221a6480a8ffa5f470b4905633821d"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.334116 5315 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:35:08.354784 5315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:35:08.357107 5315 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:35:08.357290 5315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:35:08.357539 5315 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:35:08.357961 5315 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:35:08.358052 5315 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.358121 5315 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:35:08.358176 5315 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.364501 5315 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:36565
I20260319 16:35:08.364960 5315 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:35:08.372963 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 5315
I20260319 16:35:08.373080 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:35:08.373284 25327 external_mini_cluster.cc:1468] Setting key c20c69440a881627b5c8163aa61527e3
I20260319 16:35:08.367367 5447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:36565 every 8 connection(s)
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:35:08.380405 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:08.368102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:42443 (local address 127.24.187.254:43137)
0319 16:35:08.368166 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:35:08.368168 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:35:08.368411 (+ 243us) server_negotiation.cc:408] Connection header received
0319 16:35:08.369565 (+ 1154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:35:08.369567 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:35:08.369583 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:35:08.369612 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:35:08.370583 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.371375 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:35:08.372141 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.372323 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:35:08.377428 (+ 5105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:35:08.377433 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:35:08.377434 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:35:08.377441 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:35:08.378995 (+ 1554us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:35:08.379467 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:35:08.379469 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:35:08.379469 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:35:08.379495 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:35:08.379700 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:35:08.379701 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:35:08.379702 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:35:08.379818 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:35:08.379856 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0319 16:35:08.380192 (+ 336us) server_negotiation.cc:300] Negotiation successful
0319 16:35:08.380289 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:35:08.381114 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:08.367871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:42443)
0319 16:35:08.368292 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:08.368335 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0319 16:35:08.369452 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:35:08.369797 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:35:08.369803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:35:08.370057 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:35:08.370471 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:35:08.370475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.371482 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:35:08.371482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:35:08.372049 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:35:08.372051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:35:08.372457 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:35:08.373792 (+ 1335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:35:08.373815 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:35:08.377316 (+ 3501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:35:08.379085 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:35:08.379094 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:35:08.379097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:35:08.379383 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:35:08.379553 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:35:08.379554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:35:08.379554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:35:08.379637 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:35:08.379896 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:35:08.379901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:35:08.380047 (+ 146us) client_negotiation.cc:769] Sending connection context
0319 16:35:08.380775 (+ 728us) client_negotiation.cc:240] Negotiation successful
0319 16:35:08.380904 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":68,"threads_started":1}
I20260319 16:35:08.381791 5448 heartbeater.cc:344] Connected to a master server at 127.24.187.254:43137
I20260319 16:35:08.381880 5448 heartbeater.cc:461] Registering TS with master...
I20260319 16:35:08.382082 5448 heartbeater.cc:507] Master 127.24.187.254:43137 requested a full tablet report, sending...
I20260319 16:35:08.382475 5083 ts_manager.cc:194] Registered new tserver with Master: 6cc451d324234a6c8537c65242bee2ba (127.24.187.194:36565)
I20260319 16:35:08.382982 5083 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:42443
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:35:08.445838 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:46889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:43137
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.24.187.254 from keytab /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:878)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:745)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20260319 16:35:08.458135 5170 server.cc:273] Received an EOF from the subprocess
F20260319 16:35:08.459177 5143 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fa00bbaddcd google::LogMessage::Fail() at ??:0
@ 0x7fa00bbb1b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa00bbad7cc google::LogMessage::Flush() at ??:0
@ 0x7fa00bbaef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa00cfb8783 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fa00cce01b7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa00cfd76db start_thread at ??:0
@ 0x7fa00b10261f clone at ??:0
W20260319 16:35:08.529230 5455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:35:08.529397 5455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:35:08.529426 5455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:35:08.530908 5455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:35:08.530952 5455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:35:08.530984 5455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:35:08.532580 5455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.187.212:46889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:43137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:35:08.532774 5455 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:35:08.532930 5455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:35:08.535198 5469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:35:08.535259 5455 server_base.cc:1060] running on GCE node
W20260319 16:35:08.535303 5470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:35:08.535180 5472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:35:08.535573 5455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:35:08.535758 5455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:35:08.536890 5455 hybrid_clock.cc:648] HybridClock initialized: now 1773938108536874 us; error 29 us; skew 500 ppm
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:35:08.539309 5455 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:35:08.539788 5455 webserver.cc:492] Webserver started at http://127.24.187.195:38087/ using document root <none> and password file <none>
I20260319 16:35:08.539973 5455 fs_manager.cc:362] Metadata directory not provided
I20260319 16:35:08.540011 5455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:35:08.540109 5455 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:35:08 dist-test-slave-svn8 krb5kdc[4325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938108, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:35:08.553783 5455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "264f7a698a194182a908e9620eb19580"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "06a833cee7614a68ef3f56d0ffb54661"
server_key_iv: "ecb14137742b45ba8c119d52cd45a3b2"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.554263 5455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "264f7a698a194182a908e9620eb19580"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "06a833cee7614a68ef3f56d0ffb54661"
server_key_iv: "ecb14137742b45ba8c119d52cd45a3b2"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.556834 5455 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260319 16:35:08.571599 5480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.571866 5455 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260319 16:35:08.571935 5455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "264f7a698a194182a908e9620eb19580"
format_stamp: "Formatted at 2026-03-19 16:35:08 on dist-test-slave-svn8"
server_key: "06a833cee7614a68ef3f56d0ffb54661"
server_key_iv: "ecb14137742b45ba8c119d52cd45a3b2"
server_key_version: "kuduclusterkey@0"
I20260319 16:35:08.572018 5455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:35:08.591006 5455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:35:08.593091 5455 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:35:08.593288 5455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:35:08.593503 5455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:35:08.593914 5455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:35:08.593999 5455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.594071 5455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:35:08.594126 5455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:35:08.601080 5455 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:37915
I20260319 16:35:08.601517 5455 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:35:08.603048 5593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:37915 every 8 connection(s)
I20260319 16:35:08.604219 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 5455
I20260319 16:35:08.604341 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:35:08.604499 25327 external_mini_cluster.cc:1468] Setting key 2c8219e4cd4b6042c5157cfad59f6c4b
W20260319 16:35:08.656368 4323 connection.cc:537] client connection to 127.24.187.254:43137 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260319 16:35:08.656827 5596 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:35:08.603904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:52857)
0319 16:35:08.604041 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:08.604073 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:35:08.605174 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:35:08.656644 (+ 51470us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
W20260319 16:35:08.657092 5594 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:43137 (0 consecutive failures): Network error: Failed to ping master at 127.24.187.254:43137: Client connection negotiation failed: client connection to 127.24.187.254:43137: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260319 16:35:08.657456 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:08.657313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:39903)
0319 16:35:08.657400 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:08.657435 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260319 16:35:08.657840 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:08.657773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:37379)
0319 16:35:08.657802 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:08.657826 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260319 16:35:08.657941 5594 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260319 16:35:09.121774 5307 heartbeater.cc:499] Master 127.24.187.254:43137 was elected leader, sending a full tablet report...
I20260319 16:35:09.122256 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:09.122020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:55255)
0319 16:35:09.122162 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:09.122211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
W20260319 16:35:09.122479 5307 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:43137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
I20260319 16:35:09.383944 5448 heartbeater.cc:499] Master 127.24.187.254:43137 was elected leader, sending a full tablet report...
I20260319 16:35:09.384560 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:09.384259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:59735)
0319 16:35:09.384441 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:09.384503 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
W20260319 16:35:09.384804 5448 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:43137 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
I20260319 16:35:09.658505 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:09.658294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:50995)
0319 16:35:09.658411 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:09.658439 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260319 16:35:10.122984 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:10.122761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:57835)
0319 16:35:10.122883 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:10.122920 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260319 16:35:10.385406 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:10.385165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:55707)
0319 16:35:10.385303 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:10.385338 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260319 16:35:10.659152 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:10.658928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:51059)
0319 16:35:10.659050 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:10.659086 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260319 16:35:11.123725 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:11.123485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:38597)
0319 16:35:11.123639 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:11.123681 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260319 16:35:11.386111 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:11.385910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:57185)
0319 16:35:11.386028 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:11.386063 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260319 16:35:11.659955 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:11.659732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:56577)
0319 16:35:11.659871 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:11.659907 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260319 16:35:12.124512 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:12.124247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:38245)
0319 16:35:12.124426 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:12.124467 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260319 16:35:12.386801 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:12.386543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:33549)
0319 16:35:12.386713 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:12.386753 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260319 16:35:12.660660 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:12.660425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:59291)
0319 16:35:12.660571 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:12.660610 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260319 16:35:13.125229 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:13.125024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:37483)
0319 16:35:13.125144 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:13.125185 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260319 16:35:13.387511 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:13.387281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:49561)
0319 16:35:13.387427 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:13.387464 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260319 16:35:13.661353 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:13.661110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:49713)
0319 16:35:13.661264 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:13.661303 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260319 16:35:14.126011 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:14.125743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:49103)
0319 16:35:14.125902 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:14.125949 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260319 16:35:14.388223 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:14.388001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:58567)
0319 16:35:14.388138 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:14.388175 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260319 16:35:14.662053 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:14.661851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:42187)
0319 16:35:14.661968 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:14.662005 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260319 16:35:15.126708 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:15.126469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:46013)
0319 16:35:15.126606 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:15.126663 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260319 16:35:15.389025 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:15.388775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:45747)
0319 16:35:15.388935 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:15.388977 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260319 16:35:15.662747 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:15.662517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:47823)
0319 16:35:15.662662 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:15.662700 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260319 16:35:16.127473 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:16.127241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:50291)
0319 16:35:16.127380 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:16.127426 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260319 16:35:16.389777 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:16.389551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:47179)
0319 16:35:16.389665 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:16.389702 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260319 16:35:16.663610 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:16.663373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:46825)
0319 16:35:16.663502 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:16.663535 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260319 16:35:17.128139 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:17.127912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:59643)
0319 16:35:17.128057 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:17.128095 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260319 16:35:17.390435 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:17.390220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:56817)
0319 16:35:17.390331 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:17.390363 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260319 16:35:17.664404 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:17.664140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:59731)
0319 16:35:17.664279 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:17.664353 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260319 16:35:18.128850 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:18.128628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:51313)
0319 16:35:18.128765 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:18.128805 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260319 16:35:18.391078 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:18.390844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:55651)
0319 16:35:18.390993 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:18.391029 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260319 16:35:18.665091 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:18.664854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:59039)
0319 16:35:18.665003 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:18.665041 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260319 16:35:19.129587 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:19.129382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:54103)
0319 16:35:19.129509 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:19.129540 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260319 16:35:19.391759 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:19.391543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:50543)
0319 16:35:19.391666 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:19.391713 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260319 16:35:19.665879 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:19.665573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:43249)
0319 16:35:19.665771 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:19.665817 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":115,"threads_started":1}
I20260319 16:35:20.130291 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:20.130042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:58467)
0319 16:35:20.130201 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:20.130247 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260319 16:35:20.392482 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:20.392233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:54031)
0319 16:35:20.392396 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:20.392434 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260319 16:35:20.666625 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:20.666379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:35611)
0319 16:35:20.666513 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:20.666554 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260319 16:35:21.131131 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:21.130899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:52403)
0319 16:35:21.131039 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:21.131075 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260319 16:35:21.393169 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:21.392928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:53975)
0319 16:35:21.393069 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:21.393108 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260319 16:35:21.667330 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:21.667095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:45213)
0319 16:35:21.667240 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:21.667281 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260319 16:35:22.131855 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:22.131601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:48981)
0319 16:35:22.131771 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:22.131808 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260319 16:35:22.393872 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:22.393652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:43025)
0319 16:35:22.393796 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:22.393829 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260319 16:35:22.668048 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:22.667822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.195:51351)
0319 16:35:22.667961 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:22.668000 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260319 16:35:23.132664 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:23.132420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.193:57489)
0319 16:35:23.132577 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:23.132621 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260319 16:35:23.394510 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:35:23.394301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43137 (local address 127.24.187.194:38525)
0319 16:35:23.394440 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:35:23.394466 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":112,"threads_started":1}
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:1052: Failure
Failed
Bad status: Timed out: Timed out waiting for 3 TS(s) to register with all masters
I20260319 16:35:23.615546 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 5178
I20260319 16:35:23.619886 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 5315
I20260319 16:35:23.624002 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 5455
I20260319 16:35:23.628260 25327 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260319 16:35:23.962425 25327 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260319 16:35:23.962536 25327 mini_ranger.cc:66] Stopping Ranger...
I20260319 16:35:24.306953 25327 mini_ranger.cc:68] Stopped Ranger
2026-03-19 16:35:24.307 UTC [4356] LOG: received smart shutdown request
2026-03-19 16:35:24.310 UTC [4356] LOG: background worker "logical replication launcher" (PID 4364) exited with exit code 1
2026-03-19 16:35:24.310 UTC [4359] LOG: shutting down
2026-03-19 16:35:24.312 UTC [4359] LOG: checkpoint starting: shutdown immediate
2026-03-19 16:35:25.242 UTC [4359] LOG: checkpoint complete: wrote 2418 buffers (14.8%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.033 s, sync=0.890 s, total=0.932 s; sync files=1113, longest=0.006 s, average=0.001 s; distance=12108 kB, estimate=12108 kB; lsn=0/204DC80, redo lsn=0/204DC80
2026-03-19 16:35:25.249 UTC [4356] LOG: database system is shut down
2026-03-19T16:35:25Z chronyd exiting
I20260319 16:35:25.269454 25327 test_util.cc:183] -----------------------------------------------
I20260319 16:35:25.269518 25327 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1773937967898507-25327-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (69467 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:35:25 dist-test-slave-svn8 krb5kdc[5651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:35:25 dist-test-slave-svn8 krb5kdc[5651](info): set up 2 sockets
Mar 19 16:35:25 dist-test-slave-svn8 krb5kdc[5651](info): commencing operation
krb5kdc: starting...
W20260319 16:35:27.090103 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.820s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:35:27 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:35:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:35:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/rangeradmin_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/rangeradmin_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/rangerlookup_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/rangerlookup_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
I20260319 16:35:27.244040 25327 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskq2KZu_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/postgres -l logfile start
2026-03-19 16:35:30.243 UTC [5682] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-03-19 16:35:30.243 UTC [5682] LOG: listening on IPv4 address "127.24.187.212", port 43857
2026-03-19 16:35:30.246 UTC [5682] LOG: listening on Unix socket "/tmp/.s.PGSQL.43857"
2026-03-19 16:35:30.252 UTC [5687] LOG: database system was shut down at 2026-03-19 16:35:28 UTC
2026-03-19 16:35:30.256 UTC [5682] LOG: database system is ready to accept connections
I20260319 16:35:32.009920 25327 mini_postgres.cc:96] Postgres bound to 43857
2026-03-19 16:35:32.016 UTC [5693] FATAL: database "slave" does not exist
127.24.187.212:43857 - accepting connections
I20260319 16:35:32.016633 25327 mini_ranger.cc:161] Starting Ranger...
I20260319 16:35:32.035526 25327 mini_ranger.cc:84] Created miniranger Postgres user
I20260319 16:35:32.100397 25327 mini_ranger.cc:87] Created ranger Postgres database
I20260319 16:35:32.100474 25327 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-admin
2026-03-19 16:35:32,385 [I] DB FLAVOR :POSTGRES
2026-03-19 16:35:32,385 [I] --------- Verifying Ranger DB connection ---------
2026-03-19 16:35:32,385 [I] Checking connection..
2026-03-19 16:35:32,385 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-03-19 16:35:32,729 [I] Checking connection passed.
2026-03-19 16:35:32,730 [I] --------- Verifying version history table ---------
2026-03-19 16:35:32,730 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-19 16:35:33,044 [I] Table x_db_version_h does not exist in database ranger
2026-03-19 16:35:33,044 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-19 16:35:33,361 [I] Table x_db_version_h does not exist in database ranger
2026-03-19 16:35:33,362 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-03-19 16:35:33,362 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-03-19 16:35:33.669 UTC [5806] WARNING: there is no transaction in progress
2026-03-19 16:35:33,693 [I] create_dbversion_catalog.sql file imported successfully
2026-03-19 16:35:33,693 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-03-19 16:35:34,005 [I] Table x_db_version_h already exists in database 'ranger'
2026-03-19 16:35:34,006 [I] --------- Importing Ranger Core DB Schema ---------
2026-03-19 16:35:34,006 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-03-19 16:35:34,326 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-03-19 16:35:34,656 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-svn8.c.gcp-upstream.internal','N') ;"
2026-03-19 16:35:34,970 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-03-19 16:35:35,281 [I] Table x_portal_user does not exist in database ranger
2026-03-19 16:35:35,282 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-03-19 16:35:35,590 [I] Table x_policy_ref_group does not exist in database ranger
2026-03-19 16:35:35,590 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-03-19 16:35:35,938 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-03-19 16:35:36,292 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-03-19 16:35:36,292 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-03-19 16:35:37.173 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.185 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.196 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.206 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.218 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.230 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.279 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.284 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.294 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.304 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.313 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.323 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.330 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.335 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.339 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37.343 UTC [6005] WARNING: there is no transaction in progress
2026-03-19 16:35:37,761 [I] ranger_core_db_postgres.sql file imported successfully
2026-03-19 16:35:37,761 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-03-19 16:35:38,077 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-03-19 16:35:38,078 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-03-19 16:35:38,412 [I] Table x_portal_user already exists in database 'ranger'
2026-03-19 16:35:38,413 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-03-19 16:35:38,731 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-03-19 16:35:38,731 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-03-19 16:35:39,048 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-03-19 16:35:39,378 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-svn8.c.gcp-upstream.internal';"
2026-03-19 16:35:39,697 [I] CORE_DB_SCHEMA import status has been updated
2026-03-19 16:35:39,697 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-03-19 16:35:40,034 [I] DB_PATCHES have already been applied
I20260319 16:35:40.039947 25327 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-admin:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskq2KZu_/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260319 16:35:40.040021 25327 mini_ranger.cc:193] Using host: 127.24.187.212
I20260319 16:35:40.043068 25327 mini_ranger.cc:239] Ranger admin URL: http://127.24.187.212:52649
Mar 19, 2026 4:35:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Mar 19, 2026 4:35:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Mar 19, 2026 4:35:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp
Mar 19, 2026 4:35:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Mar 19, 2026 4:35:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp] .....
Mar 19, 2026 4:35:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-admin/ews/webapp].
Mar 19, 2026 4:35:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260319 16:35:42.113582 25327 mini_ranger.cc:160] Time spent starting Ranger: real 10.097s user 0.000s sys 0.010s
Mar 19, 2026 4:36:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Mar 19, 2026 4:36:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Mar 19, 2026 4:36:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Mar 19, 2026 4:36:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Mar 19, 2026 4:36:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260319 16:36:05.958213 25327 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.187.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.187.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/rangerkms_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.187.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/rangerkms_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.187.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.187.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.187.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/HTTP_127.24.187.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Mar 19 16:36:06 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938166, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260319 16:36:06.061158 25327 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260319 16:36:06.073395 25327 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260319 16:36:06.197953 25327 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260319 16:36:06.198112 25327 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms
I20260319 16:36:06.198179 25327 mini_ranger_kms.cc:227] Using postgres at 127.24.187.212:43857
2026-03-19 16:36:06,416 [I] DB FLAVOR :POSTGRES
2026-03-19 16:36:06,417 [I] --------- Verifying Ranger DB connection ---------
2026-03-19 16:36:06,417 [I] Checking connection
2026-03-19 16:36:06,417 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-03-19 16:36:06,759 [I] connection success
2026-03-19 16:36:06,759 [I] --------- Verifying Ranger DB tables ---------
2026-03-19 16:36:06,759 [I] Verifying table ranger_masterkey in database rangerkms
2026-03-19 16:36:06,760 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-03-19 16:36:07,114 [I] Table ranger_masterkey does not exist in database rangerkms
2026-03-19 16:36:07,115 [I] --------- Importing Ranger Core DB Schema ---------
2026-03-19 16:36:07,115 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-03-19 16:36:07,115 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.187.212:43857/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-03-19 16:36:07,426 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260319 16:36:07.988690 25327 mini_ranger_kms.cc:326] Created kms service
I20260319 16:36:08.222671 25327 mini_ranger_kms.cc:342] Created kudu user
I20260319 16:36:08.269454 25327 mini_ranger_kms.cc:359] Created rangerkms user
Mar 19 16:36:08 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938166, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:08.727135 25327 mini_ranger_kms.cc:400] Added ranger policy
I20260319 16:36:08.727255 25327 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms:/tmp/dist-test-taskq2KZu_/build/release/bin/postgresql.jar:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskq2KZu_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260319 16:36:08.727273 25327 mini_ranger_kms.cc:240] Using host: 127.24.187.212
I20260319 16:36:08.730408 25327 mini_ranger_kms.cc:292] Ranger KMS PID: 6309
I20260319 16:36:08.730476 25327 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.24.187.212:55213
Mar 19, 2026 4:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Mar 19, 2026 4:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Mar 19, 2026 4:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Mar 19, 2026 4:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms/ews/webapp] .....
Mar 19, 2026 4:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-kms/ews/webapp].
Mar 19, 2026 4:36:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260319 16:36:10.819681 25327 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.759s user 0.000s sys 0.185s
I20260319 16:36:10.819787 25327 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260319 16:36:10.819836 25327 mini_ranger_kms.cc:417] 127.24.187.212:55213/kms/v1/keys
Mar 19 16:36:11 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938171, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.187.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Mar 19 16:36:11 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938171, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.187.212@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
Mar 19 16:36:12 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938171, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.187.212@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
Mar 19 16:36:14 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:14.853036 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37059
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:37059
--ranger_config_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:14.969003 6376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:14.969177 6376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:14.969195 6376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:14.970690 6376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:14.970731 6376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:14.970744 6376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:14.970754 6376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:14.970764 6376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:14.972391 6376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.187.212:55213/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:37059
--ranger_config_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:14.972571 6376 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:14.972751 6376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:36:14.975286 6376 server_base.cc:1060] running on GCE node
W20260319 16:36:14.975342 6381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:14.975329 6382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:14.975533 6384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:14.975741 6376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:14.975950 6376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:14.977088 6376 hybrid_clock.cc:648] HybridClock initialized: now 1773938174977069 us; error 31 us; skew 500 ppm
Mar 19 16:36:14 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938174, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:14.979306 6376 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:14.979734 6376 webserver.cc:492] Webserver started at http://127.24.187.254:37343/ using document root <none> and password file <none>
I20260319 16:36:14.979871 6376 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:14.979898 6376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:14.979972 6376 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:36:14 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938174, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:15.033900 6376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "74f5da4fa29f6b9e28818030755fac4a"
server_key_iv: "b74f39f7756ae033a112dda2f5ee8d53"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:15.034312 6376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "74f5da4fa29f6b9e28818030755fac4a"
server_key_iv: "b74f39f7756ae033a112dda2f5ee8d53"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:15.036398 6376 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260319 16:36:15.053011 6391 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:15.053282 6376 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:15.053359 6376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "74f5da4fa29f6b9e28818030755fac4a"
server_key_iv: "b74f39f7756ae033a112dda2f5ee8d53"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:15.053442 6376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:15.091225 6376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:15.092695 6376 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:15.092805 6376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:15.096623 6376 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:37059
I20260319 16:36:15.096839 6443 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:37059 every 8 connection(s)
I20260319 16:36:15.096968 6376 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:15.097510 6444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:15.099519 6444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Bootstrap starting.
I20260319 16:36:15.100070 6444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:15.100308 6444 log.cc:826] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:15.100705 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 6376
I20260319 16:36:15.100800 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:36:15.100991 25327 external_mini_cluster.cc:1468] Setting key 5edff06588b541b402abaa1a5f758660
I20260319 16:36:15.101050 6444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: No bootstrap required, opened a new log
I20260319 16:36:15.102377 6444 raft_consensus.cc:359] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:15.102489 6444 raft_consensus.cc:385] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:15.102509 6444 raft_consensus.cc:740] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fe1d00d0e644ce81cf5ef285fbde6b, State: Initialized, Role: FOLLOWER
I20260319 16:36:15.102617 6444 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:15.102701 6444 raft_consensus.cc:399] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:36:15.102731 6444 raft_consensus.cc:493] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:36:15.102775 6444 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:15.103423 6444 raft_consensus.cc:515] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:15.103526 6444 leader_election.cc:304] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5fe1d00d0e644ce81cf5ef285fbde6b; no voters:
I20260319 16:36:15.103679 6444 leader_election.cc:290] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:36:15.103724 6449 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:15.103868 6449 raft_consensus.cc:697] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 LEADER]: Becoming Leader. State: Replica: a5fe1d00d0e644ce81cf5ef285fbde6b, State: Running, Role: LEADER
I20260319 16:36:15.103896 6444 sys_catalog.cc:565] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:15.104054 6449 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:15.104372 6451 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } } }
I20260319 16:36:15.104445 6451 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:15.105157 6456 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:15.105355 6444 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260319 16:36:15.105638 6456 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:15.105939 6450 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5fe1d00d0e644ce81cf5ef285fbde6b. Latest consensus state: current_term: 1 leader_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } } }
I20260319 16:36:15.106066 6450 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: This master's current role is: LEADER
Mar 19 16:36:15 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:15.110065 6456 catalog_manager.cc:1357] Generated new cluster ID: 54b8163ad795456d8701043dfbd15e6b
I20260319 16:36:15.110114 6456 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:36:15.112923 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:15.101555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.24.187.254:37059)
0319 16:36:15.101735 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:15.101746 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:15.101769 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:36:15.102370 (+ 601us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:15.102379 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:15.102439 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:15.102579 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:15.103493 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:15.104934 (+ 1441us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:15.106257 (+ 1323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:15.106492 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:15.108558 (+ 2066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:15.108688 (+ 130us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:15.108692 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:15.108784 (+ 92us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:15.111261 (+ 2477us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:15.111620 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:15.111623 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:15.111623 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:15.111680 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:15.111815 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:15.111815 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:15.111816 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:15.112035 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:15.112561 (+ 526us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:15.112638 (+ 77us) server_negotiation.cc:300] Negotiation successful
0319 16:36:15.112752 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260319 16:36:15.118630 6456 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:36:15.119364 6456 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:36:15.126852 6456 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Generated new TSK 0
I20260319 16:36:15.127033 6456 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:15.868924 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37059
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:15.952942 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:15.953168 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:15.953195 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:15.954787 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:15.954836 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:15.954861 6503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:36:15.956588 6503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.187.212:55213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:15.956792 6503 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:15.956961 6503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:15.959367 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:15.959357 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:15.959447 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:15.959647 6503 server_base.cc:1060] running on GCE node
I20260319 16:36:15.959815 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:15.960028 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:15.961166 6503 hybrid_clock.cc:648] HybridClock initialized: now 1773938175961146 us; error 30 us; skew 500 ppm
Mar 19 16:36:15 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938175, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:15.964069 6503 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:36:15.964651 6503 webserver.cc:492] Webserver started at http://127.24.187.193:36639/ using document root <none> and password file <none>
I20260319 16:36:15.964856 6503 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:15.964905 6503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:15.965060 6503 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:36:15 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938175, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:15.982029 6503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "97d3d733d6064c80943213f69738d5d6"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "bae637f15d8f5d5400dbf8415e6eb5b7"
server_key_iv: "613697bed7c2ab524219beaa480271b3"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:15.982543 6503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "97d3d733d6064c80943213f69738d5d6"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "bae637f15d8f5d5400dbf8415e6eb5b7"
server_key_iv: "613697bed7c2ab524219beaa480271b3"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:15.986088 6503 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260319 16:36:16.002811 6520 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.003005 6503 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:16.003079 6503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "97d3d733d6064c80943213f69738d5d6"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "bae637f15d8f5d5400dbf8415e6eb5b7"
server_key_iv: "613697bed7c2ab524219beaa480271b3"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.003180 6503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:16.027021 6503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:16.029238 6503 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:16.029419 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:16.029747 6503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:16.030174 6503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:16.030272 6503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.030360 6503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:16.030431 6503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.036862 6503 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:43755
I20260319 16:36:16.037307 6503 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:36:16.044373 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 6503
I20260319 16:36:16.044478 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:36:16.044673 25327 external_mini_cluster.cc:1468] Setting key 90cc1ddb77a5777e2af1d26b74449f9d
I20260319 16:36:16.049660 6634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:43755 every 8 connection(s)
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938175, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:16.060253 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:16.049203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37059 (local address 127.24.187.193:56465)
0319 16:36:16.050074 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:16.050106 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:16.051157 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:16.051580 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:16.051584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:16.051886 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:16.052304 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:16.052309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.053402 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:16.053403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:16.054036 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:16.054039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.054182 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:16.055053 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:16.055076 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:16.056914 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:16.058906 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:16.058912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:16.058915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:16.059183 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:16.059339 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:16.059340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:16.059340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:16.059411 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:16.059662 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:16.059666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:16.059811 (+ 145us) client_negotiation.cc:769] Sending connection context
0319 16:36:16.059913 (+ 102us) client_negotiation.cc:240] Negotiation successful
0319 16:36:16.060032 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":88,"threads_started":1}
I20260319 16:36:16.060798 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:16.049484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:56465 (local address 127.24.187.254:37059)
0319 16:36:16.051337 (+ 1853us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:16.051339 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:16.051351 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:36:16.051376 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:16.051377 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:16.051391 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:16.051450 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:16.052421 (+ 971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.053299 (+ 878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:16.054592 (+ 1293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.054794 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:16.057015 (+ 2221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:16.057019 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:16.057020 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:16.057027 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:16.058820 (+ 1793us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:16.059257 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:16.059259 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:16.059259 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:16.059285 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:16.059464 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:16.059465 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:16.059465 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:16.059578 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:16.059628 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:16.060601 (+ 973us) server_negotiation.cc:300] Negotiation successful
0319 16:36:16.060702 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1819,"thread_start_us":78,"threads_started":1}
I20260319 16:36:16.061551 6635 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37059
I20260319 16:36:16.061657 6635 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:16.061890 6635 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:16.062485 6408 ts_manager.cc:194] Registered new tserver with Master: 97d3d733d6064c80943213f69738d5d6 (127.24.187.193:43755)
I20260319 16:36:16.063476 6408 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:56465
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:16.120826 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37059
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:16.236152 6643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:16.236413 6643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:16.236449 6643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:16.238786 6643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:16.238858 6643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:16.238901 6643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:36:16.241477 6643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.187.212:55213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:16.241815 6643 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:16.242019 6643 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:16.244755 6653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:16.244967 6651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:16.245177 6650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:16.246376 6643 server_base.cc:1060] running on GCE node
I20260319 16:36:16.246594 6643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:16.246848 6643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:16.250172 6643 hybrid_clock.cc:648] HybridClock initialized: now 1773938176250152 us; error 30 us; skew 500 ppm
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:16.252334 6643 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:36:16.252842 6643 webserver.cc:492] Webserver started at http://127.24.187.194:36681/ using document root <none> and password file <none>
I20260319 16:36:16.253036 6643 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:16.253070 6643 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:16.253183 6643 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:16.280210 6643 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "797026e9824a4b50ad8ac98d071277e2"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "3153c9ba0229250ed15bcf2faef2c69e"
server_key_iv: "a7bc9c67858abc5cce1631de210d2fd5"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.280627 6643 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "797026e9824a4b50ad8ac98d071277e2"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "3153c9ba0229250ed15bcf2faef2c69e"
server_key_iv: "a7bc9c67858abc5cce1631de210d2fd5"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.282110 6643 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:16.295552 6661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.295810 6643 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:16.295884 6643 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "797026e9824a4b50ad8ac98d071277e2"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "3153c9ba0229250ed15bcf2faef2c69e"
server_key_iv: "a7bc9c67858abc5cce1631de210d2fd5"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.295969 6643 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:16.321288 6643 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:16.323436 6643 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:16.323661 6643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:16.323945 6643 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:16.324337 6643 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:16.324440 6643 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.324515 6643 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:16.324572 6643 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.331197 6643 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:38315
I20260319 16:36:16.331228 6774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:38315 every 8 connection(s)
I20260319 16:36:16.331522 6643 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:36:16.334115 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 6643
I20260319 16:36:16.334211 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:36:16.334374 25327 external_mini_cluster.cc:1468] Setting key 1b79e39028030f24fb71e50584d8ecb4
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:16.340356 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:16.332175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:44419 (local address 127.24.187.254:37059)
0319 16:36:16.332218 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:16.332220 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:16.332451 (+ 231us) server_negotiation.cc:408] Connection header received
0319 16:36:16.333340 (+ 889us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:16.333341 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:16.333358 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:16.333411 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:16.334094 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.334628 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:16.335253 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.335429 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:16.337301 (+ 1872us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:16.337306 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:16.337306 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:16.337314 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:16.338890 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:16.339378 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:16.339380 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:16.339380 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:16.339406 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:16.339657 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:16.339658 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:16.339659 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:16.339766 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:16.339808 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:16.340155 (+ 347us) server_negotiation.cc:300] Negotiation successful
0319 16:36:16.340250 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:36:16.341069 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:16.332173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37059 (local address 127.24.187.194:44419)
0319 16:36:16.332328 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:16.332377 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:16.333242 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:16.333472 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:16.333476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:16.333665 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:16.334011 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:16.334015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.334717 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:16.334718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:16.335177 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:16.335180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.335305 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:16.335772 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:16.335788 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:16.337202 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:16.338994 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:16.338999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:16.339002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:16.339269 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:16.339477 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:16.339478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:16.339479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:16.339557 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:16.339852 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:16.339857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:16.339996 (+ 139us) client_negotiation.cc:769] Sending connection context
0319 16:36:16.340761 (+ 765us) client_negotiation.cc:240] Negotiation successful
0319 16:36:16.340880 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260319 16:36:16.341761 6775 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37059
I20260319 16:36:16.341894 6775 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:16.342131 6775 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:16.342530 6408 ts_manager.cc:194] Registered new tserver with Master: 797026e9824a4b50ad8ac98d071277e2 (127.24.187.194:38315)
I20260319 16:36:16.342990 6408 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:44419
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:16.392328 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:37059
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:16.489527 6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:16.489730 6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:16.489763 6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:16.491251 6783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:16.491297 6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:16.491331 6783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:36:16.492913 6783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.187.212:55213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:16.493104 6783 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:16.493306 6783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:16.495883 6792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:16.496120 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:16.501497 6788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:16.502506 6783 server_base.cc:1060] running on GCE node
I20260319 16:36:16.502702 6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:16.502941 6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:16.510322 6783 hybrid_clock.cc:648] HybridClock initialized: now 1773938176510283 us; error 52 us; skew 500 ppm
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:16.512863 6783 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:36:16.513410 6783 webserver.cc:492] Webserver started at http://127.24.187.195:39921/ using document root <none> and password file <none>
I20260319 16:36:16.513581 6783 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:16.513617 6783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:16.513783 6783 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:16.535912 6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "b84bf148380741049e9d0d8a8537c651"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "6b44122a0f38f7cb18ba3fa182e1c440"
server_key_iv: "1b5aeab1409159a036a9248a103f5716"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.536252 6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "b84bf148380741049e9d0d8a8537c651"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "6b44122a0f38f7cb18ba3fa182e1c440"
server_key_iv: "1b5aeab1409159a036a9248a103f5716"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.558153 6783 fs_manager.cc:696] Time spent creating directory manager: real 0.022s user 0.002s sys 0.000s
I20260319 16:36:16.595012 6800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.595577 6783 fs_manager.cc:730] Time spent opening block manager: real 0.024s user 0.001s sys 0.000s
I20260319 16:36:16.595659 6783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "b84bf148380741049e9d0d8a8537c651"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "6b44122a0f38f7cb18ba3fa182e1c440"
server_key_iv: "1b5aeab1409159a036a9248a103f5716"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.595743 6783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:16.669703 6783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:16.675783 6783 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:16.675982 6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:16.676203 6783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:16.676606 6783 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:16.676688 6783 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.676751 6783 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:16.676810 6783 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.682945 6783 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:33209
I20260319 16:36:16.685915 6783 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:36:16.690124 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 6783
I20260319 16:36:16.690236 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:36:16.690423 25327 external_mini_cluster.cc:1468] Setting key 416e38002512dde13290158ba8cbee6a
I20260319 16:36:16.693522 6913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:33209 every 8 connection(s)
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:16.710772 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:16.694356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:59425 (local address 127.24.187.254:37059)
0319 16:36:16.694403 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:16.694405 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:16.694748 (+ 343us) server_negotiation.cc:408] Connection header received
0319 16:36:16.701675 (+ 6927us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:16.701677 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:16.701691 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:16.701752 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:16.703103 (+ 1351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.704019 (+ 916us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:16.704867 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.705051 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:16.707767 (+ 2716us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:16.707771 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:16.707772 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:16.707779 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:16.709432 (+ 1653us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:16.709886 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:16.709887 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:16.709888 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:16.709913 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:16.710098 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:16.710098 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:16.710099 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:16.710204 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:16.710249 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:16.710590 (+ 341us) server_negotiation.cc:300] Negotiation successful
0319 16:36:16.710679 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:36:16.711575 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:16.694110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37059 (local address 127.24.187.195:59425)
0319 16:36:16.694647 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:16.694680 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:16.701532 (+ 6852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:16.701942 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:16.701947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:16.702298 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:16.702984 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:16.702989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.704115 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:16.704116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:16.704772 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:16.704775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:16.705254 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:16.705865 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:16.705889 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:16.707668 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:16.709523 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:16.709528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:16.709531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:16.709807 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:16.709971 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:16.709972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:16.709972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:16.710042 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:16.710277 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:16.710281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:16.710440 (+ 159us) client_negotiation.cc:769] Sending connection context
0319 16:36:16.711115 (+ 675us) client_negotiation.cc:240] Negotiation successful
0319 16:36:16.711242 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":93,"threads_started":1}
I20260319 16:36:16.712193 6914 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37059
I20260319 16:36:16.712294 6914 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:16.712509 6914 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:16.712929 6408 ts_manager.cc:194] Registered new tserver with Master: b84bf148380741049e9d0d8a8537c651 (127.24.187.195:33209)
I20260319 16:36:16.713418 6408 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:59425
I20260319 16:36:16.714663 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:36:16.715307 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 6503
I20260319 16:36:16.723556 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 6643
I20260319 16:36:16.728286 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 6783
I20260319 16:36:16.733363 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 6376
I20260319 16:36:16.738950 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37059
--webserver_interface=127.24.187.254
--webserver_port=37343
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:37059
--ranger_config_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:16.812110 6917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:16.812340 6917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:16.812372 6917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:16.814661 6917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:16.814718 6917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:16.814738 6917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:16.814755 6917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:16.814780 6917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:16.816922 6917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:37059
--ranger_config_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=37343
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:16.817117 6917 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:16.817291 6917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:16.819607 6925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:16.819633 6922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:16.819636 6923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:16.819900 6917 server_base.cc:1060] running on GCE node
I20260319 16:36:16.820070 6917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:16.820267 6917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:16.821400 6917 hybrid_clock.cc:648] HybridClock initialized: now 1773938176821382 us; error 30 us; skew 500 ppm
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:16.823593 6917 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:16.824051 6917 webserver.cc:492] Webserver started at http://127.24.187.254:37343/ using document root <none> and password file <none>
I20260319 16:36:16.824203 6917 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:16.824244 6917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Mar 19 16:36:16 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938176, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:16.867324 6917 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:36:16.868103 6933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:16.868280 6917 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:16.868360 6917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "74f5da4fa29f6b9e28818030755fac4a"
server_key_iv: "b74f39f7756ae033a112dda2f5ee8d53"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:16.868729 6917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:16.883993 6917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:16.885741 6917 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:16.885859 6917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:16.890568 6917 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:37059
I20260319 16:36:16.890771 6985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:37059 every 8 connection(s)
I20260319 16:36:16.890982 6917 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:16.892014 6986 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:36:16.892540 6986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Bootstrap starting.
I20260319 16:36:16.893595 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 6917
I20260319 16:36:16.893940 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:43755
--local_ip_for_outbound_sockets=127.24.187.193
--tserver_master_addrs=127.24.187.254:37059
--webserver_port=36639
--webserver_interface=127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:16.900166 6986 log.cc:826] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:16.901578 6986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:36:16.901854 6986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Bootstrap complete.
I20260319 16:36:16.903592 6986 raft_consensus.cc:359] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:16.903785 6986 raft_consensus.cc:740] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fe1d00d0e644ce81cf5ef285fbde6b, State: Initialized, Role: FOLLOWER
I20260319 16:36:16.903908 6986 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:16.903990 6986 raft_consensus.cc:399] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:36:16.904052 6986 raft_consensus.cc:493] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:36:16.904103 6986 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 1 FOLLOWER]: Advancing to term 2
I20260319 16:36:16.904915 6986 raft_consensus.cc:515] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:16.905009 6986 leader_election.cc:304] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5fe1d00d0e644ce81cf5ef285fbde6b; no voters:
I20260319 16:36:16.905177 6986 leader_election.cc:290] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260319 16:36:16.905225 6990 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 2 FOLLOWER]: Leader election won for term 2
I20260319 16:36:16.905398 6990 raft_consensus.cc:697] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [term 2 LEADER]: Becoming Leader. State: Replica: a5fe1d00d0e644ce81cf5ef285fbde6b, State: Running, Role: LEADER
I20260319 16:36:16.905465 6986 sys_catalog.cc:565] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:16.905525 6990 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } }
I20260319 16:36:16.905823 6990 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } } }
I20260319 16:36:16.905952 6990 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:16.906112 6990 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5fe1d00d0e644ce81cf5ef285fbde6b. Latest consensus state: current_term: 2 leader_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fe1d00d0e644ce81cf5ef285fbde6b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37059 } } }
I20260319 16:36:16.906209 6990 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:16.906340 6997 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:16.906806 6997 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:16.906987 6997 catalog_manager.cc:1269] Loaded cluster ID: 54b8163ad795456d8701043dfbd15e6b
I20260319 16:36:16.907060 6997 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:36:16.907168 6986 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260319 16:36:16.908806 6997 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:36:16.909216 6997 catalog_manager.cc:6055] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Loaded TSK: 0
I20260319 16:36:16.909461 6997 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260319 16:36:17.001547 6988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:17.001891 6988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:17.001981 6988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:17.004289 6988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:17.004415 6988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:17.004498 6988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:36:17.007043 6988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:43755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=36639
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:17.007275 6988 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:17.007500 6988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:17.010228 7033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:17.010241 7034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:17.010470 7036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:17.011307 6988 server_base.cc:1060] running on GCE node
I20260319 16:36:17.011489 6988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:17.011699 6988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:17.012852 6988 hybrid_clock.cc:648] HybridClock initialized: now 1773938177012831 us; error 34 us; skew 500 ppm
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:17.016659 6988 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:36:17.017289 6988 webserver.cc:492] Webserver started at http://127.24.187.193:36639/ using document root <none> and password file <none>
I20260319 16:36:17.017504 6988 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:17.017563 6988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:17.034508 6988 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:17.045347 7044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.046587 6988 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.000s sys 0.000s
I20260319 16:36:17.046684 6988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "97d3d733d6064c80943213f69738d5d6"
format_stamp: "Formatted at 2026-03-19 16:36:15 on dist-test-slave-svn8"
server_key: "bae637f15d8f5d5400dbf8415e6eb5b7"
server_key_iv: "613697bed7c2ab524219beaa480271b3"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:17.047066 6988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:17.065372 6988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:17.067589 6988 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:17.067790 6988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:17.068017 6988 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:17.068425 6988 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:17.068521 6988 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.068629 6988 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:17.068706 6988 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.075531 6988 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:43755
I20260319 16:36:17.076004 6988 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:36:17.081393 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 6988
I20260319 16:36:17.081823 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:38315
--local_ip_for_outbound_sockets=127.24.187.194
--tserver_master_addrs=127.24.187.254:37059
--webserver_port=36681
--webserver_interface=127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:17.078474 7158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:43755 every 8 connection(s)
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:17.093493 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:17.079052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37059 (local address 127.24.187.193:36771)
0319 16:36:17.079673 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:17.079708 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:17.080839 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:17.084121 (+ 3282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:17.084126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:17.084386 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:17.084783 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:17.084788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.086540 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:17.086541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:17.087084 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:17.087087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.087216 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:17.088170 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:17.088195 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:17.089802 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:17.091955 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:17.091960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:17.091963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:17.092214 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:17.092419 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:17.092420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:17.092421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:17.092491 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:17.092910 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:17.092915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:17.093042 (+ 127us) client_negotiation.cc:769] Sending connection context
0319 16:36:17.093166 (+ 124us) client_negotiation.cc:240] Negotiation successful
0319 16:36:17.093292 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":92,"threads_started":1}
I20260319 16:36:17.094265 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:17.079369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:36771 (local address 127.24.187.254:37059)
0319 16:36:17.081024 (+ 1655us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:17.081035 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:17.081057 (+ 22us) server_negotiation.cc:408] Connection header received
0319 16:36:17.081181 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:17.081187 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:17.083788 (+ 2601us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:17.083921 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:17.084920 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.086435 (+ 1515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:17.087637 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.087859 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:17.089904 (+ 2045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:17.089931 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:17.089931 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:17.089950 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:17.091864 (+ 1914us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.092300 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.092302 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.092303 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.092356 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.092557 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.092558 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.092558 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.092801 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:17.092881 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:17.093965 (+ 1084us) server_negotiation.cc:300] Negotiation successful
0319 16:36:17.094071 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1610,"thread_start_us":74,"threads_started":1}
I20260319 16:36:17.095407 7159 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37059
I20260319 16:36:17.095495 7159 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.095700 7159 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
W20260319 16:36:17.096508 7159 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:37059 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260319 16:36:17.096586 7159 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.096678 7159 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:17.097266 7159 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.097368 7159 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
W20260319 16:36:17.097774 7159 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260319 16:36:17.195065 7163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:17.195384 7163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:17.195489 7163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:17.197873 7163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:17.197994 7163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:17.198081 7163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:36:17.200703 7163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:38315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=36681
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:17.201253 7163 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:17.201586 7163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:17.204507 7171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:17.204995 7168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:17.205228 7163 server_base.cc:1060] running on GCE node
W20260319 16:36:17.205199 7169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:17.205546 7163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:17.205834 7163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:17.207115 7163 hybrid_clock.cc:648] HybridClock initialized: now 1773938177207099 us; error 30 us; skew 500 ppm
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:17.213799 7163 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:36:17.214520 7163 webserver.cc:492] Webserver started at http://127.24.187.194:36681/ using document root <none> and password file <none>
I20260319 16:36:17.214725 7163 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:17.214780 7163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:17.236088 7163 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260319 16:36:17.238113 7179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.238374 7163 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260319 16:36:17.238443 7163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "797026e9824a4b50ad8ac98d071277e2"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "3153c9ba0229250ed15bcf2faef2c69e"
server_key_iv: "a7bc9c67858abc5cce1631de210d2fd5"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:17.238857 7163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:17.250950 7163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:17.254578 7163 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:17.254779 7163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:17.255036 7163 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:17.255415 7163 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:17.255492 7163 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.255561 7163 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:17.255616 7163 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.262590 7163 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:38315
I20260319 16:36:17.263065 7163 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:36:17.264003 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 7163
I20260319 16:36:17.264307 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.187.212:55213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:33209
--local_ip_for_outbound_sockets=127.24.187.195
--tserver_master_addrs=127.24.187.254:37059
--webserver_port=39921
--webserver_interface=127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:17.266569 7292 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:38315 every 8 connection(s)
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:17.286788 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:17.267488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:59947 (local address 127.24.187.254:37059)
0319 16:36:17.267535 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:17.267537 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:17.270020 (+ 2483us) server_negotiation.cc:408] Connection header received
0319 16:36:17.273052 (+ 3032us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:17.273054 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:17.273073 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:17.273105 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:17.274229 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.274984 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:17.277804 (+ 2820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.277996 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:17.282417 (+ 4421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:17.282423 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:17.282424 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:17.282433 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:17.284202 (+ 1769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.284747 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.284749 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.284749 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.284777 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.285987 (+ 1210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.285989 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.285989 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.286122 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:17.286178 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:17.286556 (+ 378us) server_negotiation.cc:300] Negotiation successful
0319 16:36:17.286678 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:36:17.287573 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:17.267189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37059 (local address 127.24.187.194:59947)
0319 16:36:17.269882 (+ 2693us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:17.269922 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:17.271152 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:17.273322 (+ 2170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:17.273328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:17.273672 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:17.274115 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:17.274120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.275106 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:17.275107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:17.275671 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:17.275674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.278144 (+ 2470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:17.280424 (+ 2280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:17.280458 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:17.282273 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:17.284323 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:17.284329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:17.284332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:17.284655 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:17.284926 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:17.284926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:17.284927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:17.285893 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:17.286217 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:17.286222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:17.286390 (+ 168us) client_negotiation.cc:769] Sending connection context
0319 16:36:17.287193 (+ 803us) client_negotiation.cc:240] Negotiation successful
0319 16:36:17.287364 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2648,"thread_start_us":106,"threads_started":1}
I20260319 16:36:17.288529 7293 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37059
I20260319 16:36:17.288638 7293 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.288861 7293 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
W20260319 16:36:17.289521 7293 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:37059 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260319 16:36:17.289587 7293 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.289690 7293 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:17.290131 7293 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.290228 7293 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
W20260319 16:36:17.290560 7293 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260319 16:36:17.380518 7295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:17.380828 7295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:17.380899 7295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:17.383183 7295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:17.383386 7295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:17.383473 7295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:36:17.385897 7295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.187.212:55213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:33209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=39921
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:37059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:17.386339 7295 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:17.386569 7295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:36:17.389981 7295 server_base.cc:1060] running on GCE node
W20260319 16:36:17.390244 7302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:17.393869 7304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:17.393847 7301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:17.394838 7295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:17.395102 7295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:17.398343 7295 hybrid_clock.cc:648] HybridClock initialized: now 1773938177398318 us; error 45 us; skew 500 ppm
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:17.401285 7295 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:36:17.401949 7295 webserver.cc:492] Webserver started at http://127.24.187.195:39921/ using document root <none> and password file <none>
I20260319 16:36:17.402151 7295 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:17.402205 7295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:17.427014 7295 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:17.427649 7313 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.427841 7295 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:17.427901 7295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "b84bf148380741049e9d0d8a8537c651"
format_stamp: "Formatted at 2026-03-19 16:36:16 on dist-test-slave-svn8"
server_key: "6b44122a0f38f7cb18ba3fa182e1c440"
server_key_iv: "1b5aeab1409159a036a9248a103f5716"
server_key_version: "kuduclusterkey@0"
I20260319 16:36:17.428241 7295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:17.444285 7295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:17.446596 7295 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:17.446776 7295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:17.447024 7295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:17.447420 7295 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:17.447504 7295 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.447587 7295 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:17.447649 7295 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:17.454262 7295 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:33209
I20260319 16:36:17.454727 7295 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:36:17.455130 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 7295
I20260319 16:36:17.458716 7426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:33209 every 8 connection(s)
I20260319 16:36:17.473907 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:17.455708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.24.187.254:37059)
0319 16:36:17.455753 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:17.455755 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:17.461910 (+ 6155us) server_negotiation.cc:408] Connection header received
0319 16:36:17.462000 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:17.462001 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:17.462013 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:17.462060 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:17.466719 (+ 4659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.467433 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:17.468300 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.468473 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:17.469673 (+ 1200us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:17.469678 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:17.469678 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:17.469684 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:17.471265 (+ 1581us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.473172 (+ 1907us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.473173 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.473174 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.473200 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.473358 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.473359 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.473360 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.473456 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:17.473600 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:17.473697 (+ 97us) server_negotiation.cc:300] Negotiation successful
0319 16:36:17.473809 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Mar 19 16:36:17 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:17.484021 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:17.459680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:57205 (local address 127.24.187.254:37059)
0319 16:36:17.466779 (+ 7099us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:17.466781 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:17.471199 (+ 4418us) server_negotiation.cc:408] Connection header received
0319 16:36:17.471224 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:17.471225 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:17.471238 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:17.471303 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:17.472165 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.472942 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:17.473676 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.473857 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:17.479657 (+ 5800us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:17.479662 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:17.479663 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:17.479670 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:17.481284 (+ 1614us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.481811 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.481813 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.481813 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.481839 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:17.482066 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:17.482067 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:17.482068 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:17.482180 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:17.482229 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:17.483813 (+ 1584us) server_negotiation.cc:300] Negotiation successful
0319 16:36:17.483912 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7065,"thread_start_us":64,"threads_started":1}
I20260319 16:36:17.484802 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:17.459384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:37059 (local address 127.24.187.195:57205)
0319 16:36:17.469821 (+ 10437us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:17.469872 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:17.471064 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:17.471421 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:17.471426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:17.471673 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:17.472059 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:17.472063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.473046 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:17.473047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:17.473596 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:17.473599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:17.473987 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:17.475797 (+ 1810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:17.475822 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:17.479542 (+ 3720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:17.481387 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:17.481393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:17.481396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:17.481686 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:17.481906 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:17.481907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:17.481907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:17.481995 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:17.483469 (+ 1474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:17.483475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:17.483640 (+ 165us) client_negotiation.cc:769] Sending connection context
0319 16:36:17.484459 (+ 819us) client_negotiation.cc:240] Negotiation successful
0319 16:36:17.484594 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10387,"thread_start_us":102,"threads_started":1}
I20260319 16:36:17.485451 7427 heartbeater.cc:344] Connected to a master server at 127.24.187.254:37059
I20260319 16:36:17.485556 7427 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.485790 7427 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
W20260319 16:36:17.486420 7427 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:37059 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260319 16:36:17.486487 7427 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.486574 7427 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:17.486893 7427 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:17.486991 7427 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
W20260319 16:36:17.487317 7427 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260319 16:36:18.097929 7159 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:18.098141 7159 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:18.098516 6950 ts_manager.cc:194] Registered new tserver with Master: 97d3d733d6064c80943213f69738d5d6 (127.24.187.193:43755)
I20260319 16:36:18.099619 6950 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:36771
Mar 19 16:36:18 dist-test-slave-svn8 krb5kdc[5651](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:18.290750 7293 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:18.290931 7293 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:18.291338 6950 ts_manager.cc:194] Registered new tserver with Master: 797026e9824a4b50ad8ac98d071277e2 (127.24.187.194:38315)
I20260319 16:36:18.291863 6950 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:59947
I20260319 16:36:18.487480 7427 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:18.487696 7427 heartbeater.cc:507] Master 127.24.187.254:37059 requested a full tablet report, sending...
I20260319 16:36:18.488108 6950 ts_manager.cc:194] Registered new tserver with Master: b84bf148380741049e9d0d8a8537c651 (127.24.187.195:33209)
I20260319 16:36:18.488504 6950 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:57205
I20260319 16:36:18.493762 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Mar 19 16:36:18 dist-test-slave-svn8 krb5kdc[5651](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1773938178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for HTTP/127.24.187.212@KRBTEST.COM
I20260319 16:36:18.512825 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.494620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.24.187.254:37059)
0319 16:36:18.494737 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:18.494739 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:18.494869 (+ 130us) server_negotiation.cc:408] Connection header received
0319 16:36:18.494929 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:18.494931 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:18.494943 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:18.494998 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:18.495643 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.496142 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.496719 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.496886 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.498663 (+ 1777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:18.498667 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:18.498667 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:18.498675 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:18.508127 (+ 9452us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:18.511081 (+ 2954us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:18.511083 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:18.511084 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:18.511121 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:18.512151 (+ 1030us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:18.512152 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:18.512153 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:18.512292 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:18.512378 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:18.512613 (+ 235us) server_negotiation.cc:300] Negotiation successful
0319 16:36:18.512714 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260319 16:36:18.514994 6950 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:36:18.516912 6950 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:36:18.526199 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.520573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:43755 (local address 127.0.0.1:56536)
0319 16:36:18.520837 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:18.520869 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:18.520968 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:18.523318 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:18.523321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:18.523323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:18.523646 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.523650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.525091 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.525092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:18.525936 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.525939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.525985 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.526030 (+ 45us) client_negotiation.cc:769] Sending connection context
0319 16:36:18.526068 (+ 38us) client_negotiation.cc:240] Negotiation successful
0319 16:36:18.526109 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
I20260319 16:36:18.527570 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.520656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.24.187.193:43755)
0319 16:36:18.523119 (+ 2463us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:18.523123 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:18.523136 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:36:18.523181 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:18.523183 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:18.523202 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:18.523737 (+ 535us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:18.523829 (+ 92us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.524985 (+ 1156us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.526723 (+ 1738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.527362 (+ 639us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.527432 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:18.527462 (+ 30us) server_negotiation.cc:300] Negotiation successful
0319 16:36:18.527498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2426,"thread_start_us":59,"threads_started":1}
I20260319 16:36:18.528375 7092 tablet_service.cc:1511] Processing CreateTablet for tablet e4429810b85f42a9a67d4869da47021a (DEFAULT_TABLE table=test-table [id=4114220dfdbd4a5baf6108c8d0478e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:18.528673 7092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4429810b85f42a9a67d4869da47021a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:18.530857 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.520862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:33209 (local address 127.0.0.1:32876)
0319 16:36:18.525042 (+ 4180us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:18.525054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:18.525138 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:18.525418 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:18.525420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:18.525421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:18.525656 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.525658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.527359 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.527360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:18.530631 (+ 3271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.530635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.530709 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.530714 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:18.530744 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:36:18.530785 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4135,"thread_start_us":274,"threads_started":1}
I20260319 16:36:18.532599 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.520601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:38315 (local address 127.0.0.1:49978)
0319 16:36:18.521861 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:18.521872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:18.521946 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:18.523389 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:18.523389 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:18.523391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:18.526993 (+ 3602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.526996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.529215 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.529216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:18.532365 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.532370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.532447 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.532452 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:18.532481 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:36:18.532522 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"thread_start_us":119,"threads_started":1}
I20260319 16:36:18.532802 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.520995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.24.187.195:33209)
0319 16:36:18.521123 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:18.521127 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:18.525216 (+ 4089us) server_negotiation.cc:408] Connection header received
0319 16:36:18.525261 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:18.525263 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:18.525280 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:18.525337 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:18.525755 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.527235 (+ 1480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.530969 (+ 3734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.531682 (+ 713us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.531734 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:18.531769 (+ 35us) server_negotiation.cc:300] Negotiation successful
0319 16:36:18.531807 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260319 16:36:18.533390 7361 tablet_service.cc:1511] Processing CreateTablet for tablet e4429810b85f42a9a67d4869da47021a (DEFAULT_TABLE table=test-table [id=4114220dfdbd4a5baf6108c8d0478e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:18.533660 7361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4429810b85f42a9a67d4869da47021a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:18.533960 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.520797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.24.187.194:38315)
0319 16:36:18.523158 (+ 2361us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:18.523162 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:18.523171 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:36:18.523204 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:18.523205 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:18.523225 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:18.523288 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:18.527100 (+ 3812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.529115 (+ 2015us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.533056 (+ 3941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.533687 (+ 631us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.533782 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:18.533815 (+ 33us) server_negotiation.cc:300] Negotiation successful
0319 16:36:18.533857 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2331,"thread_start_us":43,"threads_started":1}
I20260319 16:36:18.536653 7464 tablet_bootstrap.cc:492] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Bootstrap starting.
I20260319 16:36:18.537902 7464 tablet_bootstrap.cc:654] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:18.538151 7226 tablet_service.cc:1511] Processing CreateTablet for tablet e4429810b85f42a9a67d4869da47021a (DEFAULT_TABLE table=test-table [id=4114220dfdbd4a5baf6108c8d0478e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:18.538275 7464 log.cc:826] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:18.538434 7226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4429810b85f42a9a67d4869da47021a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:18.538977 7464 tablet_bootstrap.cc:492] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: No bootstrap required, opened a new log
I20260319 16:36:18.539036 7464 ts_tablet_manager.cc:1403] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:36:18.541020 7464 raft_consensus.cc:359] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.541150 7464 raft_consensus.cc:385] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:18.541175 7464 raft_consensus.cc:740] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84bf148380741049e9d0d8a8537c651, State: Initialized, Role: FOLLOWER
I20260319 16:36:18.541278 7464 consensus_queue.cc:260] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.541538 7464 ts_tablet_manager.cc:1434] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260319 16:36:18.541590 7427 heartbeater.cc:499] Master 127.24.187.254:37059 was elected leader, sending a full tablet report...
I20260319 16:36:18.542660 7467 tablet_bootstrap.cc:492] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Bootstrap starting.
I20260319 16:36:18.543303 7467 tablet_bootstrap.cc:654] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:18.543665 7467 log.cc:826] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:18.544459 7467 tablet_bootstrap.cc:492] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: No bootstrap required, opened a new log
I20260319 16:36:18.544533 7467 ts_tablet_manager.cc:1403] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:18.546505 7467 raft_consensus.cc:359] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.546715 7467 raft_consensus.cc:385] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:18.546766 7467 raft_consensus.cc:740] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797026e9824a4b50ad8ac98d071277e2, State: Initialized, Role: FOLLOWER
I20260319 16:36:18.546877 7467 consensus_queue.cc:260] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.547137 7467 ts_tablet_manager.cc:1434] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260319 16:36:18.547174 7293 heartbeater.cc:499] Master 127.24.187.254:37059 was elected leader, sending a full tablet report...
I20260319 16:36:18.558583 7463 tablet_bootstrap.cc:492] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6: Bootstrap starting.
I20260319 16:36:18.559530 7463 tablet_bootstrap.cc:654] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:18.559842 7463 log.cc:826] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:18.560904 7463 tablet_bootstrap.cc:492] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6: No bootstrap required, opened a new log
I20260319 16:36:18.561044 7463 ts_tablet_manager.cc:1403] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260319 16:36:18.563148 7463 raft_consensus.cc:359] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.563310 7463 raft_consensus.cc:385] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:18.563345 7463 raft_consensus.cc:740] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d3d733d6064c80943213f69738d5d6, State: Initialized, Role: FOLLOWER
I20260319 16:36:18.563525 7463 consensus_queue.cc:260] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.563815 7463 ts_tablet_manager.cc:1434] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260319 16:36:18.563861 7159 heartbeater.cc:499] Master 127.24.187.254:37059 was elected leader, sending a full tablet report...
W20260319 16:36:18.580485 7160 tablet.cc:2379] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:36:18.660259 7469 raft_consensus.cc:493] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:36:18.660414 7469 raft_consensus.cc:515] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.660805 7469 leader_election.cc:290] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97d3d733d6064c80943213f69738d5d6 (127.24.187.193:43755), b84bf148380741049e9d0d8a8537c651 (127.24.187.195:33209)
I20260319 16:36:18.663640 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.661242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:33209 (local address 127.24.187.194:55369)
0319 16:36:18.661378 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:18.661387 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:18.661451 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:18.661636 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:18.661637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:18.661639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:18.661981 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.661984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.662793 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.662794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:18.663416 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.663419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.663487 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.663492 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:18.663521 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:36:18.663557 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260319 16:36:18.664048 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.661310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:55369 (local address 127.24.187.195:33209)
0319 16:36:18.661352 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:18.661354 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:18.661451 (+ 97us) server_negotiation.cc:408] Connection header received
0319 16:36:18.661508 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:18.661509 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:18.661519 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:18.661577 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:18.662070 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.662723 (+ 653us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.663496 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.663943 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.663952 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:18.663976 (+ 24us) server_negotiation.cc:300] Negotiation successful
0319 16:36:18.664003 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:36:18.664316 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.661090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:38411 (local address 127.24.187.193:43755)
0319 16:36:18.661154 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:18.661156 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:18.661290 (+ 134us) server_negotiation.cc:408] Connection header received
0319 16:36:18.661454 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:18.661455 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:18.661466 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:18.661518 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:18.661852 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.662520 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.663622 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.664070 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.664080 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:18.664204 (+ 124us) server_negotiation.cc:300] Negotiation successful
0319 16:36:18.664247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:36:18.664372 7381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4429810b85f42a9a67d4869da47021a" candidate_uuid: "797026e9824a4b50ad8ac98d071277e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84bf148380741049e9d0d8a8537c651" is_pre_election: true
I20260319 16:36:18.664515 7381 raft_consensus.cc:2468] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 797026e9824a4b50ad8ac98d071277e2 in term 0.
I20260319 16:36:18.664530 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.660999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:43755 (local address 127.24.187.194:38411)
0319 16:36:18.661196 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:18.661209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:18.661322 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:18.661529 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:18.661530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:18.661531 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:18.661785 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.661788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.662626 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.662627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:18.663511 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:18.663514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.664110 (+ 596us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.664115 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:18.664426 (+ 311us) client_negotiation.cc:240] Negotiation successful
0319 16:36:18.664467 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":120,"threads_started":1}
I20260319 16:36:18.664870 7182 leader_election.cc:304] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 797026e9824a4b50ad8ac98d071277e2, b84bf148380741049e9d0d8a8537c651; no voters:
I20260319 16:36:18.664857 7110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4429810b85f42a9a67d4869da47021a" candidate_uuid: "797026e9824a4b50ad8ac98d071277e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d3d733d6064c80943213f69738d5d6" is_pre_election: true
I20260319 16:36:18.664983 7110 raft_consensus.cc:2468] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 797026e9824a4b50ad8ac98d071277e2 in term 0.
I20260319 16:36:18.664991 7469 raft_consensus.cc:2804] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:36:18.665030 7469 raft_consensus.cc:493] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:36:18.665052 7469 raft_consensus.cc:3060] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:18.665764 7469 raft_consensus.cc:515] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.665874 7469 leader_election.cc:290] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [CANDIDATE]: Term 1 election: Requested vote from peers 97d3d733d6064c80943213f69738d5d6 (127.24.187.193:43755), b84bf148380741049e9d0d8a8537c651 (127.24.187.195:33209)
I20260319 16:36:18.666015 7110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4429810b85f42a9a67d4869da47021a" candidate_uuid: "797026e9824a4b50ad8ac98d071277e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d3d733d6064c80943213f69738d5d6"
I20260319 16:36:18.666095 7110 raft_consensus.cc:3060] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:18.666075 7381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4429810b85f42a9a67d4869da47021a" candidate_uuid: "797026e9824a4b50ad8ac98d071277e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84bf148380741049e9d0d8a8537c651"
I20260319 16:36:18.666152 7381 raft_consensus.cc:3060] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:18.666749 7110 raft_consensus.cc:2468] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 797026e9824a4b50ad8ac98d071277e2 in term 1.
I20260319 16:36:18.666924 7182 leader_election.cc:304] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 797026e9824a4b50ad8ac98d071277e2, 97d3d733d6064c80943213f69738d5d6; no voters:
I20260319 16:36:18.666998 7469 raft_consensus.cc:2804] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:18.667058 7381 raft_consensus.cc:2468] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 797026e9824a4b50ad8ac98d071277e2 in term 1.
I20260319 16:36:18.667140 7469 raft_consensus.cc:697] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 1 LEADER]: Becoming Leader. State: Replica: 797026e9824a4b50ad8ac98d071277e2, State: Running, Role: LEADER
I20260319 16:36:18.667236 7469 consensus_queue.cc:237] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } }
I20260319 16:36:18.668054 6949 catalog_manager.cc:5671] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 797026e9824a4b50ad8ac98d071277e2 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "797026e9824a4b50ad8ac98d071277e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "797026e9824a4b50ad8ac98d071277e2" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 } health_report { overall_health: UNKNOWN } } }
I20260319 16:36:18.679529 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.676815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.24.187.194:38315)
0319 16:36:18.676856 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:18.676858 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:18.676870 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:36:18.676937 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:18.676938 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:18.676959 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:18.677018 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:36:18.677347 (+ 329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.678128 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:18.678938 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:18.679121 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:18.679166 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:36:18.679283 (+ 117us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:36:18.679364 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:18.679426 (+ 62us) server_negotiation.cc:300] Negotiation successful
0319 16:36:18.679451 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:36:18.681110 7381 raft_consensus.cc:1275] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 1 FOLLOWER]: Refusing update from remote peer 797026e9824a4b50ad8ac98d071277e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:18.681166 7110 raft_consensus.cc:1275] T e4429810b85f42a9a67d4869da47021a P 97d3d733d6064c80943213f69738d5d6 [term 1 FOLLOWER]: Refusing update from remote peer 797026e9824a4b50ad8ac98d071277e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:18.681465 7469 consensus_queue.cc:1048] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d3d733d6064c80943213f69738d5d6" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:18.681630 7469 consensus_queue.cc:1048] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84bf148380741049e9d0d8a8537c651" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:18.687347 7480 mvcc.cc:204] Tried to move back new op lower bound from 7266050779875270656 to 7266050779821441024. Current Snapshot: MvccSnapshot[applied={T|T < 7266050779875270656}]
I20260319 16:36:18.688557 7478 mvcc.cc:204] Tried to move back new op lower bound from 7266050779875270656 to 7266050779821441024. Current Snapshot: MvccSnapshot[applied={T|T < 7266050779875270656}]
I20260319 16:36:18.690985 6949 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:18.691082 6949 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:18.691859 6949 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a5fe1d00d0e644ce81cf5ef285fbde6b: Sending DeleteTablet for 3 replicas of tablet e4429810b85f42a9a67d4869da47021a
I20260319 16:36:18.692341 7092 tablet_service.cc:1558] Processing DeleteTablet for tablet e4429810b85f42a9a67d4869da47021a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:18 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:56536
I20260319 16:36:18.692409 7226 tablet_service.cc:1558] Processing DeleteTablet for tablet e4429810b85f42a9a67d4869da47021a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:18 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:49978
I20260319 16:36:18.692565 7361 tablet_service.cc:1558] Processing DeleteTablet for tablet e4429810b85f42a9a67d4869da47021a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:18 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:32876
I20260319 16:36:18.692603 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 6988
I20260319 16:36:18.694728 7488 tablet_replica.cc:333] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: stopping tablet replica
I20260319 16:36:18.694816 7488 raft_consensus.cc:2243] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:36:18.694938 7488 raft_consensus.cc:2272] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:18.695073 7489 tablet_replica.cc:333] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: stopping tablet replica
I20260319 16:36:18.698369 7488 ts_tablet_manager.cc:1916] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:18.700093 7488 ts_tablet_manager.cc:1929] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:18.700228 7488 log.cc:1199] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/e4429810b85f42a9a67d4869da47021a
W20260319 16:36:18.700232 6936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:43755 (error 108)
I20260319 16:36:18.700609 7488 ts_tablet_manager.cc:1950] T e4429810b85f42a9a67d4869da47021a P 797026e9824a4b50ad8ac98d071277e2: Deleting consensus metadata
I20260319 16:36:18.700940 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 7163
I20260319 16:36:18.701428 6935 catalog_manager.cc:5002] TS 797026e9824a4b50ad8ac98d071277e2 (127.24.187.194:38315): tablet e4429810b85f42a9a67d4869da47021a (table test-table [id=4114220dfdbd4a5baf6108c8d0478e4b]) successfully deleted
I20260319 16:36:18.695147 7489 raft_consensus.cc:2243] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:36:18.703969 7489 raft_consensus.cc:2272] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:18.704341 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.704178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:43755 (local address 127.0.0.1:56548)
0319 16:36:18.704239 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:18.704294 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260319 16:36:18.704460 6936 catalog_manager.cc:4729] TS 97d3d733d6064c80943213f69738d5d6 (127.24.187.193:43755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4429810b85f42a9a67d4869da47021a: Network error: Client connection negotiation failed: client connection to 127.24.187.193:43755: connect: Connection refused (error 111)
I20260319 16:36:18.704917 7489 ts_tablet_manager.cc:1916] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:18.706504 7489 ts_tablet_manager.cc:1929] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:18.706600 7489 log.cc:1199] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/e4429810b85f42a9a67d4869da47021a
I20260319 16:36:18.706921 7489 ts_tablet_manager.cc:1950] T e4429810b85f42a9a67d4869da47021a P b84bf148380741049e9d0d8a8537c651: Deleting consensus metadata
I20260319 16:36:18.707010 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 7295
I20260319 16:36:18.713907 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:18.713781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:33209 (local address 127.0.0.1:32892)
0319 16:36:18.713841 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:18.713867 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.195:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260319 16:36:18.713883 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 6917
I20260319 16:36:18.719409 25327 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260319 16:36:19.053614 25327 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260319 16:36:19.053737 25327 mini_ranger.cc:66] Stopping Ranger...
I20260319 16:36:19.398536 25327 mini_ranger.cc:68] Stopped Ranger
2026-03-19 16:36:19.398 UTC [5682] LOG: received smart shutdown request
2026-03-19 16:36:19.402 UTC [5682] LOG: background worker "logical replication launcher" (PID 5690) exited with exit code 1
2026-03-19 16:36:19.403 UTC [5685] LOG: shutting down
2026-03-19 16:36:19.404 UTC [5685] LOG: checkpoint starting: shutdown immediate
2026-03-19 16:36:20.446 UTC [5685] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.035 s, sync=1.000 s, total=1.044 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12073 kB, estimate=12073 kB; lsn=0/2044CF0, redo lsn=0/2044CF0
2026-03-19 16:36:20.453 UTC [5682] LOG: database system is shut down
2026-03-19T16:36:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55296 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:36:20 dist-test-slave-svn8 krb5kdc[7499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:36:20 dist-test-slave-svn8 krb5kdc[7499](info): set up 2 sockets
Mar 19 16:36:20 dist-test-slave-svn8 krb5kdc[7499](info): commencing operation
krb5kdc: starting...
W20260319 16:36:22.402187 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.835s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:36:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:36:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:22.533008 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36167
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:36167,127.24.187.253:39361,127.24.187.252:33521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:22.607367 7515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:22.607530 7515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:22.607553 7515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:22.609017 7515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:22.609063 7515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:22.609076 7515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:22.609087 7515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:22.609097 7515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:22.610757 7515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:36167,127.24.187.253:39361,127.24.187.252:33521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:22.610958 7515 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:22.611125 7515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:22.613272 7521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:22.613272 7520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:22.613286 7523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:22.613387 7515 server_base.cc:1060] running on GCE node
I20260319 16:36:22.613698 7515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:22.613950 7515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:22.615094 7515 hybrid_clock.cc:648] HybridClock initialized: now 1773938182615078 us; error 28 us; skew 500 ppm
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:22.617244 7515 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:22.617770 7515 webserver.cc:492] Webserver started at http://127.24.187.254:37247/ using document root <none> and password file <none>
I20260319 16:36:22.617929 7515 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:22.617964 7515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:22.618064 7515 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:22.618952 7515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "18873d4c90d9408b85c3dbd19f70e0ad"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "f38cb6c45c7654c8255f562afd504d71"
server_key_iv: "498d2b12c732b52eb4ecafae97192258"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.619235 7515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "18873d4c90d9408b85c3dbd19f70e0ad"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "f38cb6c45c7654c8255f562afd504d71"
server_key_iv: "498d2b12c732b52eb4ecafae97192258"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.620299 7515 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:36:22.620939 7530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:22.621104 7515 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:36:22.621160 7515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "18873d4c90d9408b85c3dbd19f70e0ad"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "f38cb6c45c7654c8255f562afd504d71"
server_key_iv: "498d2b12c732b52eb4ecafae97192258"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.621223 7515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:22.629089 7515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:22.630910 7515 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:22.631027 7515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:22.634795 7515 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:36167
I20260319 16:36:22.634831 7582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:36167 every 8 connection(s)
I20260319 16:36:22.635123 7515 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:22.635661 7583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:22.636921 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 7515
I20260319 16:36:22.637007 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:36:22.637182 25327 external_mini_cluster.cc:1468] Setting key d9a69cee765c7ee20f757c00d77a675b
I20260319 16:36:22.637465 7583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.187.253@KRBTEST.COM; defaulting to no policy
I20260319 16:36:22.645968 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.638137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.24.187.254:36167)
0319 16:36:22.638295 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.638313 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.638401 (+ 88us) server_negotiation.cc:408] Connection header received
0319 16:36:22.638856 (+ 455us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.638858 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.638884 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.638949 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.639683 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.640735 (+ 1052us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.641260 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.641493 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.643136 (+ 1643us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.643145 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.643145 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.643152 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.644704 (+ 1552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.645027 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.645030 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.645030 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.645073 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.645280 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.645282 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.645282 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.645450 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.645524 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.645690 (+ 166us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.645867 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260319 16:36:22.645972 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.638124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36167 (local address 127.0.0.1:50546)
0319 16:36:22.638295 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.638324 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.638718 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.638969 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.638971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.639338 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.639577 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.639580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.640824 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.640825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.641182 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.641183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.641291 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.641766 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.641781 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.643053 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.644782 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.644787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.644789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.644943 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.645157 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.645158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.645158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.645205 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.645520 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.645524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.645581 (+ 57us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.645663 (+ 82us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.645801 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260319 16:36:22.646721 7583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.253@KRBTEST.COM" created.
I20260319 16:36:22.647128 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.647007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46072)
0319 16:36:22.647055 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.647104 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260319 16:36:22.647243 7534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
I20260319 16:36:22.647656 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.647570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46086)
0319 16:36:22.647612 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.647637 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260319 16:36:22.647745 7583 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:39361: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:22.681145 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:39361
--webserver_interface=127.24.187.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:36167,127.24.187.253:39361,127.24.187.252:33521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:22.696954 7583 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } attempt: 1
I20260319 16:36:22.697440 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.697298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46092)
0319 16:36:22.697362 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.697408 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260319 16:36:22.697791 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.697696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46106)
0319 16:36:22.697744 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.697769 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260319 16:36:22.697875 7583 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:39361: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
W20260319 16:36:22.754889 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:22.755050 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:22.755070 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:22.756458 7592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:22.756492 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:22.756505 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:22.756515 7592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:22.756525 7592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:22.758106 7592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:36167,127.24.187.253:39361,127.24.187.252:33521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:39361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.187.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:22.758270 7592 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:22.758401 7592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:22.760538 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:22.760565 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:22.760555 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:22.760733 7592 server_base.cc:1060] running on GCE node
I20260319 16:36:22.760927 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:22.761111 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:22.762244 7592 hybrid_clock.cc:648] HybridClock initialized: now 1773938182762221 us; error 32 us; skew 500 ppm
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:22.764463 7592 init.cc:377] Logged in from keytab as kudu/127.24.187.253@KRBTEST.COM (short username kudu)
I20260319 16:36:22.764937 7592 webserver.cc:492] Webserver started at http://127.24.187.253:46363/ using document root <none> and password file <none>
I20260319 16:36:22.765075 7592 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:22.765105 7592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:22.765182 7592 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:22.765991 7583 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } attempt: 2
I20260319 16:36:22.766124 7592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data/instance:
uuid: "3b751c3652a74813b88dcb1cfdb7e8c0"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "51fa02c322da89d89d12d3f09a386d50"
server_key_iv: "f33e93c8ae1bde3e17c6e98546988535"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.766311 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.766206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46120)
0319 16:36:22.766254 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.766279 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260319 16:36:22.766445 7592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/wal/instance:
uuid: "3b751c3652a74813b88dcb1cfdb7e8c0"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "51fa02c322da89d89d12d3f09a386d50"
server_key_iv: "f33e93c8ae1bde3e17c6e98546988535"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.766628 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.766548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46124)
0319 16:36:22.766573 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.766596 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260319 16:36:22.766698 7583 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:39361: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39361: connect: Connection refused (error 111)
I20260319 16:36:22.767563 7592 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:36:22.768164 7607 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:22.768324 7592 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:36:22.768409 7592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/wal
uuid: "3b751c3652a74813b88dcb1cfdb7e8c0"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "51fa02c322da89d89d12d3f09a386d50"
server_key_iv: "f33e93c8ae1bde3e17c6e98546988535"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.768473 7592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:22.781620 7592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:22.783249 7592 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:22.783386 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:22.787096 7592 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.253:39361
I20260319 16:36:22.787145 7659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.253:39361 every 8 connection(s)
I20260319 16:36:22.787427 7592 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
I20260319 16:36:22.787930 7660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:22.789633 7660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:22.795251 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 7592
I20260319 16:36:22.795356 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-1/wal/instance
I20260319 16:36:22.795522 25327 external_mini_cluster.cc:1468] Setting key 7bd028e908f0a3f2b738f9dab012477a
I20260319 16:36:22.796885 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.790279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.24.187.254:36167)
0319 16:36:22.790372 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.790375 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.790517 (+ 142us) server_negotiation.cc:408] Connection header received
0319 16:36:22.790872 (+ 355us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.790873 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.790885 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.790937 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.791660 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.792090 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.792622 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.792788 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.794596 (+ 1808us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.794600 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.794600 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.794607 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.795850 (+ 1243us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.796164 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.796166 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.796166 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.796189 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.796339 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.796339 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.796340 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.796425 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.796480 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.796706 (+ 226us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.796792 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260319 16:36:22.796947 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.790237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36167 (local address 127.0.0.1:50550)
0319 16:36:22.790413 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.790444 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.790788 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.790999 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.791003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.791295 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.791568 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.791571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.792176 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.792176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.792544 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.792545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.792673 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.793167 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.793182 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.794500 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.795923 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.795930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.795932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.796101 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.796236 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.796236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.796237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.796289 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.796474 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.796479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.796592 (+ 113us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.796680 (+ 88us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.796797 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260319 16:36:22.797765 7660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.187.252@KRBTEST.COM; defaulting to no policy
I20260319 16:36:22.804152 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.798004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46138)
0319 16:36:22.798051 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.798063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.798106 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.798298 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.798299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.798448 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.798630 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.798632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.799532 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.799533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.800050 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.800052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.800124 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.800507 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.800511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.801748 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.803315 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.803316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.803316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.803459 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.803621 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.803622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.803622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.803659 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.803850 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.803851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.803886 (+ 35us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.803946 (+ 60us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.804038 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260319 16:36:22.804215 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.798052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.24.187.253:39361)
0319 16:36:22.798180 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.798186 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.798201 (+ 15us) server_negotiation.cc:408] Connection header received
0319 16:36:22.798220 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.798222 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.798231 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.798284 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.798737 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.799447 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.800138 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.800307 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.801832 (+ 1525us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.801839 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.801840 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.801846 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.803244 (+ 1398us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.803527 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.803530 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.803530 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.803564 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.803708 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.803709 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.803709 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.803799 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.803870 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.803971 (+ 101us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.804099 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260319 16:36:22.804811 7660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.252@KRBTEST.COM" created.
I20260319 16:36:22.805131 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.805013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37660)
0319 16:36:22.805056 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.805107 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260319 16:36:22.805243 7611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
I20260319 16:36:22.805665 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.805578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37672)
0319 16:36:22.805620 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.805648 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260319 16:36:22.805784 7660 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33521: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:22.842160 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:33521
--webserver_interface=127.24.187.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:36167,127.24.187.253:39361,127.24.187.252:33521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:22.854969 7660 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } attempt: 1
I20260319 16:36:22.855422 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.855297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37674)
0319 16:36:22.855352 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.855390 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260319 16:36:22.855753 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.855674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37684)
0319 16:36:22.855711 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.855735 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260319 16:36:22.855844 7660 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33521: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
I20260319 16:36:22.857820 7583 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } attempt: 3
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:36:22.863556 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.858081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46146)
0319 16:36:22.858131 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.858141 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.858187 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.858311 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.858313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.858488 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.858656 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.858658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.859259 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.859260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.859603 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.859604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.859662 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.860119 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.860123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.861284 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.862866 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.862867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.862868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.862994 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.863130 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.863131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.863131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.863151 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.863336 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.863337 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.863376 (+ 39us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.863424 (+ 48us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.863493 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260319 16:36:22.863610 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.858140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.24.187.253:39361)
0319 16:36:22.858193 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.858195 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.858221 (+ 26us) server_negotiation.cc:408] Connection header received
0319 16:36:22.858247 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.858248 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.858258 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.858300 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.858732 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.859181 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.859674 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.859842 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.861359 (+ 1517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.861362 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.861363 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.861370 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.862792 (+ 1422us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.863054 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.863055 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.863055 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.863078 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.863187 (+ 109us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.863188 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.863188 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.863288 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.863347 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.863448 (+ 101us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.863526 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:36:22.863940 7583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:36:22.864238 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.864151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37694)
0319 16:36:22.864190 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.864216 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260319 16:36:22.864513 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.864457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37702)
0319 16:36:22.864480 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.864500 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260319 16:36:22.864585 7583 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33521: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
I20260319 16:36:22.895735 7583 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } attempt: 1
I20260319 16:36:22.896199 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.896068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37712)
0319 16:36:22.896114 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.896162 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260319 16:36:22.896576 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.896475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37716)
0319 16:36:22.896523 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.896551 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260319 16:36:22.896675 7583 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33521: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
W20260319 16:36:22.917464 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:22.917630 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:22.917658 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:22.919173 7669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:22.919219 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:22.919239 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:22.919253 7669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:22.919268 7669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:22.920892 7669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:36167,127.24.187.253:39361,127.24.187.252:33521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:33521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.187.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:22.921097 7669 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:22.921250 7669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:22.923449 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:22.923496 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:22.923609 7669 server_base.cc:1060] running on GCE node
W20260319 16:36:22.923555 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:22.923820 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:22.923967 7660 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } attempt: 2
I20260319 16:36:22.924027 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:22.924311 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.924203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37722)
0319 16:36:22.924254 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.924280 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:36:22.924587 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.924511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37730)
0319 16:36:22.924544 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.924564 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260319 16:36:22.924666 7660 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33521: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33521: connect: Connection refused (error 111)
I20260319 16:36:22.925153 7669 hybrid_clock.cc:648] HybridClock initialized: now 1773938182925133 us; error 35 us; skew 500 ppm
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:22.927500 7669 init.cc:377] Logged in from keytab as kudu/127.24.187.252@KRBTEST.COM (short username kudu)
I20260319 16:36:22.928015 7669 webserver.cc:492] Webserver started at http://127.24.187.252:35287/ using document root <none> and password file <none>
I20260319 16:36:22.928155 7669 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:22.928184 7669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:22.928261 7669 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:22.929131 7669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data/instance:
uuid: "342144ec7a7044e9943bb765a0b0b907"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "1682293a4e41b8468834d91f318d49cf"
server_key_iv: "206c6373480daa2b0ff2f80db7d5943b"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.929380 7669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/wal/instance:
uuid: "342144ec7a7044e9943bb765a0b0b907"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "1682293a4e41b8468834d91f318d49cf"
server_key_iv: "206c6373480daa2b0ff2f80db7d5943b"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.930480 7669 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:36:22.931090 7684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:22.931241 7669 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:36:22.931294 7669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/wal
uuid: "342144ec7a7044e9943bb765a0b0b907"
format_stamp: "Formatted at 2026-03-19 16:36:22 on dist-test-slave-svn8"
server_key: "1682293a4e41b8468834d91f318d49cf"
server_key_iv: "206c6373480daa2b0ff2f80db7d5943b"
server_key_version: "encryptionkey@0"
I20260319 16:36:22.931360 7669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:22.946519 7669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:22.948175 7669 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:22.948275 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:22.952015 7669 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.252:33521
I20260319 16:36:22.952049 7737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.252:33521 every 8 connection(s)
I20260319 16:36:22.952315 7669 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
I20260319 16:36:22.952832 7738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:22.954490 7738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:36:22.956468 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 7669
I20260319 16:36:22.956570 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1773937967898507-25327-0/minicluster-data/master-2/wal/instance
I20260319 16:36:22.956732 25327 external_mini_cluster.cc:1468] Setting key 3ca80310646b926ca21ef3351ba763e5
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:36:22.962157 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.955081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.24.187.254:36167)
0319 16:36:22.955152 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.955154 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.955268 (+ 114us) server_negotiation.cc:408] Connection header received
0319 16:36:22.955622 (+ 354us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.955623 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.955633 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.955682 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.956333 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.956946 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.957628 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.957814 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.959659 (+ 1845us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.959663 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.959664 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.959672 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.960978 (+ 1306us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.961316 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.961318 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.961318 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.961342 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.961562 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.961563 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.961563 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.961665 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.961758 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.962027 (+ 269us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.962092 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260319 16:36:22.962309 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.955035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36167 (local address 127.0.0.1:50566)
0319 16:36:22.955174 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.955199 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.955549 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.955733 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.955736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.956007 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.956264 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.956267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.957096 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.957098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.957541 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.957543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.957670 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.958155 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.958170 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.959565 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.961049 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.961055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.961058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.961244 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.961413 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.961413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.961414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.961490 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.961748 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.961753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.961890 (+ 137us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.962014 (+ 124us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.962130 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
I20260319 16:36:22.962852 7738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:36:22.964113 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.957107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.24.187.252:33521)
0319 16:36:22.957266 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.957274 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.957291 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:36:22.957309 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.957310 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.957323 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.957378 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.958103 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.958802 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.959417 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.959613 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.961442 (+ 1829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.961449 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.961449 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.961454 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.963102 (+ 1648us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.963438 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.963442 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.963442 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.963475 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.963686 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.963687 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.963688 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.963790 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.963887 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.963990 (+ 103us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.964051 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:22.968731 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.963093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39361 (local address 127.0.0.1:46162)
0319 16:36:22.963137 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.963147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.963180 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.963297 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.963298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.963411 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.963564 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.963566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.964313 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.964314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.964687 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.964690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.964751 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.965077 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.965080 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.966436 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.967984 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.967984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.967985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.968104 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.968246 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.968247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.968247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.968266 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.968475 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.968476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.968514 (+ 38us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.968567 (+ 53us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.968651 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260319 16:36:22.968755 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.963152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.24.187.253:39361)
0319 16:36:22.963192 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.963194 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.963201 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:36:22.963226 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.963227 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.963236 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.963291 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.963667 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.964247 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.964777 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.964939 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.966515 (+ 1576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.966518 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.966519 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.966525 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.967909 (+ 1384us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.968165 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.968166 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.968166 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.968187 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.968327 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.968328 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.968329 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.968421 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.968478 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.968595 (+ 117us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.968675 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:36:22.969039 7738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:36:22.971505 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.965095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.24.187.254:36167)
0319 16:36:22.965156 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.965158 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.965192 (+ 34us) server_negotiation.cc:408] Connection header received
0319 16:36:22.965259 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.965260 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.965269 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.965326 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.966035 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.966595 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.967388 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.967503 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.969099 (+ 1596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.969102 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.969103 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.969109 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.970499 (+ 1390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.970766 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.970767 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.970767 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.970786 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.971092 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.971093 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.971094 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.971174 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.971298 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.971335 (+ 37us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.971421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260319 16:36:22.971791 7583 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } attempt: 2
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:36:22.976008 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.969257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37752)
0319 16:36:22.969297 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.969307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.969343 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.969470 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.969471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.969602 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.969762 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.969764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.970361 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.970361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.970682 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.970683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.970738 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.971030 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.971034 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.972447 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.975237 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.975238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.975238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.975366 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.975546 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.975547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.975548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.975569 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.975749 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.975750 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.975783 (+ 33us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.975834 (+ 51us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.975924 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:36:22.976588 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.969323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.24.187.252:33521)
0319 16:36:22.969362 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.969364 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.969371 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:36:22.969411 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.969412 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.969419 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.969462 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.969825 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.970301 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.970741 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.970891 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.972532 (+ 1641us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.972535 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.972536 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.972542 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.975162 (+ 2620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.975474 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.975475 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.975475 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.975494 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.975615 (+ 121us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.975616 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.975616 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.975701 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.975752 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.975857 (+ 105us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.976510 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":573,"server-negotiator.queue_time_us":23}
I20260319 16:36:22.978147 7738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907: Bootstrap starting.
I20260319 16:36:22.978814 7738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:22.978904 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.972066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.24.187.253:39361)
0319 16:36:22.972101 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.972103 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.972110 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:36:22.972188 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.972189 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.972198 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.972229 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.972883 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.973344 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.973944 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.974089 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.976457 (+ 2368us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.976460 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.976460 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.976466 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.977873 (+ 1407us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.978132 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.978133 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.978134 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.978154 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.978473 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.978474 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.978474 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.978548 (+ 74us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.978675 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.978745 (+ 70us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.978825 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:36:22.979096 7738 log.cc:826] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:22.979126 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.971984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37760)
0319 16:36:22.972095 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:22.972106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:22.972145 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:22.972308 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:22.972309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:22.972455 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:22.972589 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.972590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.973341 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.973342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:22.973665 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:22.973666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.973972 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.974482 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:22.974486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:22.975637 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:22.978272 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.978274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.978274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.978409 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.978574 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:22.978575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:22.978575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:22.978609 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:22.978857 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:22.978858 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:22.978905 (+ 47us) client_negotiation.cc:769] Sending connection context
0319 16:36:22.978953 (+ 48us) client_negotiation.cc:240] Negotiation successful
0319 16:36:22.979044 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260319 16:36:22.979452 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:22.972038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.24.187.252:33521)
0319 16:36:22.972177 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:22.972178 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:22.972188 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:36:22.972204 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:22.972204 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:22.972212 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:22.972296 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:22.972658 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.973269 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:22.973772 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:22.973953 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:22.975729 (+ 1776us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:22.975732 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:22.975733 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:22.975739 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:22.978166 (+ 2427us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.978473 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.978474 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.978474 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.978499 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:22.978671 (+ 172us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:22.978671 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:22.978672 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:22.978780 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:22.979225 (+ 445us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:22.979267 (+ 42us) server_negotiation.cc:300] Negotiation successful
0319 16:36:22.979344 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260319 16:36:22.979928 7738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907: No bootstrap required, opened a new log
I20260319 16:36:22.980477 7583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: Bootstrap starting.
I20260319 16:36:22.981055 7738 raft_consensus.cc:359] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:22.981150 7583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:22.981158 7738 raft_consensus.cc:385] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:22.981185 7738 raft_consensus.cc:740] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342144ec7a7044e9943bb765a0b0b907, State: Initialized, Role: FOLLOWER
I20260319 16:36:22.981281 7738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:22.981384 7583 log.cc:826] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:22.981557 7738 sys_catalog.cc:565] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:22.981542 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:22.981770 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:36:22.982049 7583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: No bootstrap required, opened a new log
I20260319 16:36:22.982334 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:36:22.983448 7583 raft_consensus.cc:359] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:22.983564 7583 raft_consensus.cc:385] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:22.983597 7583 raft_consensus.cc:740] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18873d4c90d9408b85c3dbd19f70e0ad, State: Initialized, Role: FOLLOWER
I20260319 16:36:22.983682 7583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:22.983904 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:22.983947 7583 sys_catalog.cc:565] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:22.984023 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [sys.catalog]: This master's current role is: FOLLOWER
W20260319 16:36:22.983726 7757 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:36:22.984781 7757 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260319 16:36:22.985802 7771 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:36:22.985908 7771 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:36:22.985183 7669 master_runner.cc:186] Error getting master registration for 127.24.187.254:36167: OK, instance_id { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" instance_seqno: 1773938182633695 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260319 16:36:22.987313 7669 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260319 16:36:22.986363 7515 master_runner.cc:186] Error getting master registration for 127.24.187.253:39361: OK, instance_id { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" instance_seqno: 1773938182786028 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260319 16:36:22.987972 7515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Mar 19 16:36:22 dist-test-slave-svn8 krb5kdc[7499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260319 16:36:23.016191 7660 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } attempt: 3
Mar 19 16:36:23 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
Mar 19 16:36:23 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
Mar 19 16:36:23 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
Mar 19 16:36:23 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:36:23.023211 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:23.010914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.24.187.253:39361)
0319 16:36:23.010964 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:23.010966 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:23.011520 (+ 554us) server_negotiation.cc:408] Connection header received
0319 16:36:23.011623 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:23.011625 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:23.011638 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:23.011668 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:23.012555 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.013257 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:23.014054 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.014228 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:23.018590 (+ 4362us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:23.018593 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:23.018594 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:23.018611 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:23.020928 (+ 2317us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.022480 (+ 1552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.022482 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.022482 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.022505 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.022656 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.022656 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.022657 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.022774 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:23.022842 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:23.023035 (+ 193us) server_negotiation.cc:300] Negotiation successful
0319 16:36:23.023109 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260319 16:36:23.023370 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:23.011077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.24.187.254:36167)
0319 16:36:23.011120 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:23.011123 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:23.011240 (+ 117us) server_negotiation.cc:408] Connection header received
0319 16:36:23.011351 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:23.011352 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:23.011362 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:23.011392 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:23.012338 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.013044 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:23.013856 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.014030 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:23.019273 (+ 5243us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:23.019277 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:23.019277 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:23.019284 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:23.021813 (+ 2529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.022609 (+ 796us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.022611 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.022611 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.022650 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.022820 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.022821 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.022822 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.022926 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:23.022992 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:23.023198 (+ 206us) server_negotiation.cc:300] Negotiation successful
0319 16:36:23.023271 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:36:23.025465 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:23.011250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.24.187.252:33521)
0319 16:36:23.011291 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:23.011293 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:23.014520 (+ 3227us) server_negotiation.cc:408] Connection header received
0319 16:36:23.014617 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:23.014619 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:23.014630 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:23.014659 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:23.015401 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.016125 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:23.017456 (+ 1331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.017639 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:23.022270 (+ 4631us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:23.022273 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:23.022273 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:23.022279 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:23.024338 (+ 2059us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.024680 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.024681 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.024681 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.024701 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.024892 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.024893 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.024893 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.024994 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:23.025061 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:23.025247 (+ 186us) server_negotiation.cc:300] Negotiation successful
0319 16:36:23.025325 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:36:23.025944 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:23.016503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33521 (local address 127.0.0.1:37778)
0319 16:36:23.016637 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:23.016648 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:23.016684 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:23.016954 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:23.016956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:23.017111 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:23.017279 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:23.017281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.018137 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:23.018138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:23.018631 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:23.018633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.018691 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:23.019579 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:23.019584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:23.022110 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:23.025055 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:23.025055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:23.025056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:23.025216 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:23.025372 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:23.025372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:23.025373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:23.025401 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:23.025615 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:23.025615 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:23.025671 (+ 56us) client_negotiation.cc:769] Sending connection context
0319 16:36:23.025775 (+ 104us) client_negotiation.cc:240] Negotiation successful
0319 16:36:23.025861 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260319 16:36:23.026273 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:23.016755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.24.187.252:33521)
0319 16:36:23.016810 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:23.016812 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:23.016821 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:36:23.016841 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:23.016842 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:23.016851 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:23.016902 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:23.017369 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.018048 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:23.018834 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:23.019012 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:23.022175 (+ 3163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:23.022178 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:23.022179 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:23.022185 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:23.024981 (+ 2796us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.025280 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.025281 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.025281 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.025313 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:23.025455 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:23.025456 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:23.025456 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:23.025537 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:23.025588 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:23.026119 (+ 531us) server_negotiation.cc:300] Negotiation successful
0319 16:36:23.026195 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:36:23.027431 7660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: Bootstrap starting.
I20260319 16:36:23.028190 7660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:23.028439 7660 log.cc:826] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:23.029240 7660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: No bootstrap required, opened a new log
I20260319 16:36:23.031024 7660 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:23.031132 7660 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:23.031162 7660 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b751c3652a74813b88dcb1cfdb7e8c0, State: Initialized, Role: FOLLOWER
I20260319 16:36:23.031271 7660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:23.031603 7660 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:23.031580 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.031734 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [sys.catalog]: This master's current role is: FOLLOWER
W20260319 16:36:23.033767 7793 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:36:23.033900 7613 tablet.cc:2379] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:36:23.034125 7793 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:36:23.035467 7592 master_runner.cc:263] No leader master found from master 3b751c3652a74813b88dcb1cfdb7e8c0
I20260319 16:36:23.035508 7592 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260319 16:36:23.131732 7536 tablet.cc:2379] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:36:23.198884 7691 tablet.cc:2379] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:36:23.447249 7745 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:36:23.447386 7745 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:23.447641 7745 leader_election.cc:290] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18873d4c90d9408b85c3dbd19f70e0ad (127.24.187.254:36167), 3b751c3652a74813b88dcb1cfdb7e8c0 (127.24.187.253:39361)
I20260319 16:36:23.447876 7557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342144ec7a7044e9943bb765a0b0b907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" is_pre_election: true
I20260319 16:36:23.447939 7634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342144ec7a7044e9943bb765a0b0b907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" is_pre_election: true
I20260319 16:36:23.448017 7557 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342144ec7a7044e9943bb765a0b0b907 in term 0.
I20260319 16:36:23.448092 7634 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342144ec7a7044e9943bb765a0b0b907 in term 0.
I20260319 16:36:23.448187 7686 leader_election.cc:304] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18873d4c90d9408b85c3dbd19f70e0ad, 342144ec7a7044e9943bb765a0b0b907; no voters:
I20260319 16:36:23.448329 7745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:36:23.448379 7745 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:36:23.448400 7745 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:23.449090 7745 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:23.449230 7745 leader_election.cc:290] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [CANDIDATE]: Term 1 election: Requested vote from peers 18873d4c90d9408b85c3dbd19f70e0ad (127.24.187.254:36167), 3b751c3652a74813b88dcb1cfdb7e8c0 (127.24.187.253:39361)
I20260319 16:36:23.449415 7557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342144ec7a7044e9943bb765a0b0b907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18873d4c90d9408b85c3dbd19f70e0ad"
I20260319 16:36:23.449419 7634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342144ec7a7044e9943bb765a0b0b907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0"
I20260319 16:36:23.449488 7557 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:23.449491 7634 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:23.450224 7557 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342144ec7a7044e9943bb765a0b0b907 in term 1.
I20260319 16:36:23.450225 7634 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342144ec7a7044e9943bb765a0b0b907 in term 1.
I20260319 16:36:23.450418 7689 leader_election.cc:304] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342144ec7a7044e9943bb765a0b0b907, 3b751c3652a74813b88dcb1cfdb7e8c0; no voters:
I20260319 16:36:23.450522 7745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:23.450629 7745 raft_consensus.cc:697] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [term 1 LEADER]: Becoming Leader. State: Replica: 342144ec7a7044e9943bb765a0b0b907, State: Running, Role: LEADER
I20260319 16:36:23.450704 7745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } }
I20260319 16:36:23.451014 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342144ec7a7044e9943bb765a0b0b907. Latest consensus state: current_term: 1 leader_uuid: "342144ec7a7044e9943bb765a0b0b907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.451085 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:23.451290 7797 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:23.451464 7797 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:23.452602 7557 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [term 1 FOLLOWER]: Refusing update from remote peer 342144ec7a7044e9943bb765a0b0b907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:23.452632 7634 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [term 1 FOLLOWER]: Refusing update from remote peer 342144ec7a7044e9943bb765a0b0b907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:23.452849 7795 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:23.452948 7745 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:23.453933 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342144ec7a7044e9943bb765a0b0b907. Latest consensus state: current_term: 1 leader_uuid: "342144ec7a7044e9943bb765a0b0b907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.453949 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342144ec7a7044e9943bb765a0b0b907. Latest consensus state: current_term: 1 leader_uuid: "342144ec7a7044e9943bb765a0b0b907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.454010 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:36:23.454022 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:36:23.454809 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "342144ec7a7044e9943bb765a0b0b907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.454882 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:23.454859 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "342144ec7a7044e9943bb765a0b0b907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.454911 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:23.454965 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "342144ec7a7044e9943bb765a0b0b907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.455049 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:36:23.455027 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "342144ec7a7044e9943bb765a0b0b907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18873d4c90d9408b85c3dbd19f70e0ad" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36167 } } peers { permanent_uuid: "3b751c3652a74813b88dcb1cfdb7e8c0" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39361 } } peers { permanent_uuid: "342144ec7a7044e9943bb765a0b0b907" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33521 } } }
I20260319 16:36:23.455087 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:36:23.455193 7800 mvcc.cc:204] Tried to move back new op lower bound from 7266050799419949056 to 7266050799414591488. Current Snapshot: MvccSnapshot[applied={T|T < 7266050799419949056}]
I20260319 16:36:23.455230 7799 mvcc.cc:204] Tried to move back new op lower bound from 7266050799419949056 to 7266050799414591488. Current Snapshot: MvccSnapshot[applied={T|T < 7266050799419949056}]
I20260319 16:36:23.455493 7797 catalog_manager.cc:1357] Generated new cluster ID: 393f6e4274f2402296eb8b3eed672206
I20260319 16:36:23.455542 7797 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:36:23.484550 7797 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:36:23.485244 7797 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:36:23.502111 7797 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 342144ec7a7044e9943bb765a0b0b907: Generated new TSK 0
I20260319 16:36:23.502444 7797 catalog_manager.cc:1524] Initializing in-progress tserver states...
Mar 19 16:36:23 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.24.187.252@KRBTEST.COM
Mar 19 16:36:23 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.24.187.254@KRBTEST.COM
Mar 19 16:36:23 dist-test-slave-svn8 krb5kdc[7499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.24.187.253@KRBTEST.COM
I20260319 16:36:23.986320 7771 catalog_manager.cc:1269] Loaded cluster ID: 393f6e4274f2402296eb8b3eed672206
I20260319 16:36:23.986392 7771 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: loading cluster ID for follower catalog manager: success
I20260319 16:36:23.987255 7771 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: acquiring CA information for follower catalog manager: success
I20260319 16:36:23.987536 7771 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 18873d4c90d9408b85c3dbd19f70e0ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260319 16:36:24.034574 7793 catalog_manager.cc:1269] Loaded cluster ID: 393f6e4274f2402296eb8b3eed672206
I20260319 16:36:24.034648 7793 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: loading cluster ID for follower catalog manager: success
I20260319 16:36:24.035544 7793 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: acquiring CA information for follower catalog manager: success
I20260319 16:36:24.035817 7793 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3b751c3652a74813b88dcb1cfdb7e8c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260319 16:36:24.076128 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 7515
I20260319 16:36:24.081003 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 7592
I20260319 16:36:24.085486 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 7669
2026-03-19T16:36:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3536 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:36:24 dist-test-slave-svn8 krb5kdc[7812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:36:24 dist-test-slave-svn8 krb5kdc[7812](info): set up 2 sockets
Mar 19 16:36:24 dist-test-slave-svn8 krb5kdc[7812](info): commencing operation
krb5kdc: starting...
W20260319 16:36:25.922134 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.818s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:36:25 dist-test-slave-svn8 krb5kdc[7812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:36:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:36:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:26.056218 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37925
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:37925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:26.129806 7828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:26.129992 7828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:26.130023 7828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:26.131549 7828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:26.131595 7828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:26.131614 7828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:26.131630 7828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:26.131644 7828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:26.133354 7828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:37925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:26.133585 7828 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:26.133776 7828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:26.136116 7833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:26.136111 7834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:26.136107 7836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:26.136128 7828 server_base.cc:1060] running on GCE node
I20260319 16:36:26.136458 7828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:26.136675 7828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:26.137784 7828 hybrid_clock.cc:648] HybridClock initialized: now 1773938186137768 us; error 29 us; skew 500 ppm
Mar 19 16:36:26 dist-test-slave-svn8 krb5kdc[7812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938186, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:26.140168 7828 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:26.140695 7828 webserver.cc:492] Webserver started at http://127.24.187.254:45893/ using document root <none> and password file <none>
I20260319 16:36:26.140856 7828 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:26.140889 7828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:26.140990 7828 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:26.141937 7828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "c0927c3998fc4d1099d69bdad7d0636c"
format_stamp: "Formatted at 2026-03-19 16:36:26 on dist-test-slave-svn8"
server_key: "93f5ffa07190fedae532d43411026285"
server_key_iv: "e93e978ae4012b30637f644dba40320e"
server_key_version: "encryptionkey@0"
I20260319 16:36:26.142230 7828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "c0927c3998fc4d1099d69bdad7d0636c"
format_stamp: "Formatted at 2026-03-19 16:36:26 on dist-test-slave-svn8"
server_key: "93f5ffa07190fedae532d43411026285"
server_key_iv: "e93e978ae4012b30637f644dba40320e"
server_key_version: "encryptionkey@0"
I20260319 16:36:26.143342 7828 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:26.143961 7843 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:26.144126 7828 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:36:26.144183 7828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "c0927c3998fc4d1099d69bdad7d0636c"
format_stamp: "Formatted at 2026-03-19 16:36:26 on dist-test-slave-svn8"
server_key: "93f5ffa07190fedae532d43411026285"
server_key_iv: "e93e978ae4012b30637f644dba40320e"
server_key_version: "encryptionkey@0"
I20260319 16:36:26.144245 7828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:26.167435 7828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:26.172955 7828 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:26.173089 7828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:26.177031 7828 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:37925
I20260319 16:36:26.177063 7895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:37925 every 8 connection(s)
I20260319 16:36:26.177361 7828 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:26.177940 7896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:26.180320 7896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c: Bootstrap starting.
I20260319 16:36:26.180866 7896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:26.181025 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 7828
I20260319 16:36:26.181124 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:36:26.181145 7896 log.cc:826] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:26.181298 25327 external_mini_cluster.cc:1468] Setting key b9dfd58a5bbad4f0cf18fe1e3b2848af
I20260319 16:36:26.181929 7896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c: No bootstrap required, opened a new log
I20260319 16:36:26.183159 7896 raft_consensus.cc:359] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0927c3998fc4d1099d69bdad7d0636c" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37925 } }
I20260319 16:36:26.183274 7896 raft_consensus.cc:385] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:26.183300 7896 raft_consensus.cc:740] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0927c3998fc4d1099d69bdad7d0636c, State: Initialized, Role: FOLLOWER
I20260319 16:36:26.183391 7896 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0927c3998fc4d1099d69bdad7d0636c" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37925 } }
I20260319 16:36:26.183452 7896 raft_consensus.cc:399] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:36:26.183481 7896 raft_consensus.cc:493] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:36:26.183525 7896 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:26.184028 7896 raft_consensus.cc:515] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0927c3998fc4d1099d69bdad7d0636c" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37925 } }
I20260319 16:36:26.184126 7896 leader_election.cc:304] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0927c3998fc4d1099d69bdad7d0636c; no voters:
I20260319 16:36:26.184252 7896 leader_election.cc:290] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:36:26.184314 7901 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:26.184439 7896 sys_catalog.cc:565] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:26.184458 7901 raft_consensus.cc:697] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [term 1 LEADER]: Becoming Leader. State: Replica: c0927c3998fc4d1099d69bdad7d0636c, State: Running, Role: LEADER
I20260319 16:36:26.184731 7901 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0927c3998fc4d1099d69bdad7d0636c" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37925 } }
I20260319 16:36:26.185685 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0927c3998fc4d1099d69bdad7d0636c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0927c3998fc4d1099d69bdad7d0636c" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37925 } } }
I20260319 16:36:26.185909 7902 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0927c3998fc4d1099d69bdad7d0636c. Latest consensus state: current_term: 1 leader_uuid: "c0927c3998fc4d1099d69bdad7d0636c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0927c3998fc4d1099d69bdad7d0636c" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 37925 } } }
I20260319 16:36:26.186277 7902 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [sys.catalog]: This master's current role is: LEADER
W20260319 16:36:26.186345 7915 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:36:26.186443 7915 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Mar 19 16:36:26 dist-test-slave-svn8 krb5kdc[7812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:26.186190 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:26.187321 7918 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:26.187485 7918 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:26.190076 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:26.181764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.24.187.254:37925)
0319 16:36:26.181937 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:26.181947 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:26.181967 (+ 20us) server_negotiation.cc:408] Connection header received
0319 16:36:26.182310 (+ 343us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:26.182316 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:26.182355 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:26.182463 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:26.183172 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:26.183989 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:26.184731 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:26.185000 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:26.187279 (+ 2279us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:26.187286 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:26.187287 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:26.187305 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:26.188944 (+ 1639us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:26.189227 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:26.189230 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:26.189231 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:26.189273 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:26.189415 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:26.189416 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:26.189417 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:26.189583 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:26.189750 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:26.189836 (+ 86us) server_negotiation.cc:300] Negotiation successful
0319 16:36:26.189950 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260319 16:36:26.190902 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:36:26.198122 7918 catalog_manager.cc:1357] Generated new cluster ID: 2091f5a83acb4e79b04ad19e2c9e006b
I20260319 16:36:26.198184 7918 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:36:26 dist-test-slave-svn8 krb5kdc[7812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260319 16:36:26.205309 7918 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:36:26.205865 7918 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:36:26.210845 7918 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c0927c3998fc4d1099d69bdad7d0636c: Generated new TSK 0
I20260319 16:36:26.211022 7918 catalog_manager.cc:1524] Initializing in-progress tserver states...
Mar 19 16:36:26 dist-test-slave-svn8 krb5kdc[7812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.24.187.254@KRBTEST.COM
I20260319 16:36:26.267299 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 7828
2026-03-19T16:36:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2180 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:36:26 dist-test-slave-svn8 krb5kdc[7925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:36:26 dist-test-slave-svn8 krb5kdc[7925](info): set up 2 sockets
Mar 19 16:36:26 dist-test-slave-svn8 krb5kdc[7925](info): commencing operation
krb5kdc: starting...
W20260319 16:36:28.094305 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.809s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:36:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:28.228938 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39639
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:28.303469 7941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:28.303653 7941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:28.303674 7941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:28.305156 7941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:28.305202 7941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:28.305214 7941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:28.305225 7941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:28.305235 7941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:28.306846 7941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:28.307046 7941 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:28.307206 7941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:28.309612 7946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:28.309640 7949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.309680 7941 server_base.cc:1060] running on GCE node
W20260319 16:36:28.309595 7947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.310010 7941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:28.310217 7941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:28.311362 7941 hybrid_clock.cc:648] HybridClock initialized: now 1773938188311343 us; error 27 us; skew 500 ppm
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:28.313596 7941 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:28.314105 7941 webserver.cc:492] Webserver started at http://127.24.187.254:33401/ using document root <none> and password file <none>
I20260319 16:36:28.314268 7941 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:28.314299 7941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:28.314427 7941 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:28.315330 7941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "f8e773f5586ac879160192c194c0e75a"
server_key_iv: "496d381593d3556da4b7a70b47f5a891"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.315639 7941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "f8e773f5586ac879160192c194c0e75a"
server_key_iv: "496d381593d3556da4b7a70b47f5a891"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.316745 7941 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:36:28.317293 7956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.317466 7941 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:36:28.317533 7941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "f8e773f5586ac879160192c194c0e75a"
server_key_iv: "496d381593d3556da4b7a70b47f5a891"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.317589 7941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:28.345340 7941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:28.346943 7941 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:28.347067 7941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:28.350704 7941 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39639
I20260319 16:36:28.350740 8008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39639 every 8 connection(s)
I20260319 16:36:28.351014 7941 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:28.351524 8009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:28.353946 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Bootstrap starting.
I20260319 16:36:28.353997 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 7941
I20260319 16:36:28.354104 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:36:28.354285 25327 external_mini_cluster.cc:1468] Setting key d2cd59df7240e2533c2bb8ebbeeacd70
I20260319 16:36:28.354614 8009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:28.355072 8009 log.cc:826] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:28.355773 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: No bootstrap required, opened a new log
I20260319 16:36:28.357085 8009 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:28.357223 8009 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:28.357251 8009 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa3a6209d3f4b1396b7f1c726b2e3b6, State: Initialized, Role: FOLLOWER
I20260319 16:36:28.357342 8009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:28.357411 8009 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:36:28.357440 8009 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:36:28.357483 8009 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:28.358127 8009 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:28.358235 8009 leader_election.cc:304] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa3a6209d3f4b1396b7f1c726b2e3b6; no voters:
I20260319 16:36:28.358426 8009 leader_election.cc:290] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:36:28.358480 8014 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:28.358657 8009 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:28.358662 8014 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa3a6209d3f4b1396b7f1c726b2e3b6, State: Running, Role: LEADER
I20260319 16:36:28.358866 8014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:28.359165 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } } }
I20260319 16:36:28.359163 8016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa3a6209d3f4b1396b7f1c726b2e3b6. Latest consensus state: current_term: 1 leader_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } } }
I20260319 16:36:28.359269 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:28.359275 8016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: This master's current role is: LEADER
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:28.360337 8021 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:28.360751 8021 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:28.362056 8021 catalog_manager.cc:1357] Generated new cluster ID: be663a29f6d949d986c9f02850e0c161
I20260319 16:36:28.362107 8021 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:36:28.363585 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.354729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.24.187.254:39639)
0319 16:36:28.354886 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.354898 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.354922 (+ 24us) server_negotiation.cc:408] Connection header received
0319 16:36:28.355325 (+ 403us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.355332 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.355383 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.355482 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:28.356211 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.357319 (+ 1108us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.357966 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.358204 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.360527 (+ 2323us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:28.360534 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:28.360535 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:28.360552 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:28.362474 (+ 1922us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.362759 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.362762 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.362762 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.362803 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.362946 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.362947 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.362948 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.363108 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:28.363257 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.363324 (+ 67us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.363433 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260319 16:36:28.377522 8021 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:36:28.378100 8021 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:36:28.386299 8021 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Generated new TSK 0
I20260319 16:36:28.386521 8021 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:28.438344 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39639
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:28.512575 8037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:28.512750 8037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:28.512780 8037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:28.514285 8037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:28.514331 8037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:28.514366 8037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:36:28.515952 8037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:28.516142 8037 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:28.516297 8037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:28.518553 8042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:28.518564 8045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.518679 8037 server_base.cc:1060] running on GCE node
W20260319 16:36:28.518607 8043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.518924 8037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:28.519104 8037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:28.520238 8037 hybrid_clock.cc:648] HybridClock initialized: now 1773938188520221 us; error 26 us; skew 500 ppm
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:28.522508 8037 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:36:28.522989 8037 webserver.cc:492] Webserver started at http://127.24.187.193:39657/ using document root <none> and password file <none>
I20260319 16:36:28.523150 8037 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:28.523185 8037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:28.523290 8037 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:28.524173 8037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "3a974ef1544f2ff83744cd577d28dc3f"
server_key_iv: "136cfc55e299b726b3b971b391dc6ad8"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.524457 8037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "3a974ef1544f2ff83744cd577d28dc3f"
server_key_iv: "136cfc55e299b726b3b971b391dc6ad8"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.525486 8037 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:36:28.526135 8052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.526278 8037 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.526333 8037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "3a974ef1544f2ff83744cd577d28dc3f"
server_key_iv: "136cfc55e299b726b3b971b391dc6ad8"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.526392 8037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:28.544214 8037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:28.546070 8037 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:28.546192 8037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:28.546398 8037 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:28.546706 8037 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:28.546739 8037 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.546769 8037 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:28.546793 8037 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.552160 8037 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:42519
I20260319 16:36:28.552227 8166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:42519 every 8 connection(s)
I20260319 16:36:28.552471 8037 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:36:28.553129 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 8037
I20260319 16:36:28.553264 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:36:28.553448 25327 external_mini_cluster.cc:1468] Setting key 10bd64db7e6505d21d6ee77d5702f615
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
I20260319 16:36:28.561973 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.553238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:36575 (local address 127.24.187.254:39639)
0319 16:36:28.553290 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.553292 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.553749 (+ 457us) server_negotiation.cc:408] Connection header received
0319 16:36:28.554362 (+ 613us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.554364 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.554374 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.554435 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:28.555547 (+ 1112us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.556190 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.556782 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.556953 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.559232 (+ 2279us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:28.559235 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:28.559235 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:28.559242 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:28.560525 (+ 1283us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.560882 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.560883 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.560884 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.560916 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.561205 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.561206 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.561206 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.561348 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:28.561419 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.561820 (+ 401us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.561906 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:36:28.562045 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.553152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:36575)
0319 16:36:28.553620 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.553660 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.554230 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.554488 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.554493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.554937 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:28.555378 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.555383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.556289 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.556290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.556694 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.556696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.556812 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.557277 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:28.557294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:28.559137 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:28.560627 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:28.560632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:28.560633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:28.560813 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:28.561073 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:28.561074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:28.561075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:28.561140 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:28.561452 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:28.561459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:28.561618 (+ 159us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.561772 (+ 154us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.561888 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":77,"threads_started":1}
I20260319 16:36:28.562636 8167 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39639
I20260319 16:36:28.562726 8167 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:28.562907 8167 heartbeater.cc:507] Master 127.24.187.254:39639 requested a full tablet report, sending...
I20260319 16:36:28.563469 7973 ts_manager.cc:194] Registered new tserver with Master: 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
I20260319 16:36:28.564318 7973 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:36575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:28.600554 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39639
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:28.673409 8174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:28.673599 8174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:28.673622 8174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:28.675112 8174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:28.675156 8174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:28.675181 8174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:36:28.676787 8174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:28.676978 8174 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:28.677132 8174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:28.679421 8179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:28.679499 8180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.679575 8174 server_base.cc:1060] running on GCE node
W20260319 16:36:28.679481 8182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.679845 8174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:28.680011 8174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:28.681133 8174 hybrid_clock.cc:648] HybridClock initialized: now 1773938188681116 us; error 32 us; skew 500 ppm
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:28.683399 8174 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:36:28.683843 8174 webserver.cc:492] Webserver started at http://127.24.187.194:39543/ using document root <none> and password file <none>
I20260319 16:36:28.683982 8174 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:28.684026 8174 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:28.684123 8174 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:28.685010 8174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "cbea135349c1477c9fd86ec60a7b9f75"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "2387295af3a0aad324f22b53baf4c7ef"
server_key_iv: "476dcf620ec9b63ae1846ef581c9ade2"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.685300 8174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "cbea135349c1477c9fd86ec60a7b9f75"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "2387295af3a0aad324f22b53baf4c7ef"
server_key_iv: "476dcf620ec9b63ae1846ef581c9ade2"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.686429 8174 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:36:28.687134 8189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.687312 8174 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:28.687386 8174 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "cbea135349c1477c9fd86ec60a7b9f75"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "2387295af3a0aad324f22b53baf4c7ef"
server_key_iv: "476dcf620ec9b63ae1846ef581c9ade2"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.687435 8174 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:28.705425 8174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:28.706936 8174 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:28.707036 8174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:28.707208 8174 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:28.707494 8174 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:28.707526 8174 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.707546 8174 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:28.707557 8174 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.712777 8174 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:43453
I20260319 16:36:28.712842 8302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:43453 every 8 connection(s)
I20260319 16:36:28.713091 8174 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:36:28.714551 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 8174
I20260319 16:36:28.714654 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:36:28.714833 25327 external_mini_cluster.cc:1468] Setting key 09ad0370d98a80f90ed8017990deedc5
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:28.721482 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.713748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:50993 (local address 127.24.187.254:39639)
0319 16:36:28.713810 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.713812 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.713967 (+ 155us) server_negotiation.cc:408] Connection header received
0319 16:36:28.714468 (+ 501us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.714470 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.714482 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.714533 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:28.715363 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.715901 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.716410 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.716569 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.719097 (+ 2528us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:28.719101 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:28.719101 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:28.719108 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:28.720371 (+ 1263us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.720728 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.720729 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.720730 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.720753 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.720903 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.720904 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.720904 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.721027 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:28.721083 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.721303 (+ 220us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.721393 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:36:28.721587 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.713689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:50993)
0319 16:36:28.713839 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.713869 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.714339 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.714592 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.714597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.714957 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:28.715279 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.715284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.715977 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.715979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.716330 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.716332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.716439 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.716969 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:28.716992 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:28.719012 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:28.720446 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:28.720452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:28.720454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:28.720665 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:28.720800 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:28.720801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:28.720802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:28.720854 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:28.721079 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:28.721085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:28.721190 (+ 105us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.721279 (+ 89us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.721409 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260319 16:36:28.722164 8303 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39639
I20260319 16:36:28.722240 8303 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:28.722407 8303 heartbeater.cc:507] Master 127.24.187.254:39639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
I20260319 16:36:28.722819 7973 ts_manager.cc:194] Registered new tserver with Master: cbea135349c1477c9fd86ec60a7b9f75 (127.24.187.194:43453)
I20260319 16:36:28.723166 7973 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:50993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:28.762388 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39639
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:28.836572 8310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:28.836741 8310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:28.836769 8310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:28.838274 8310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:28.838315 8310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:28.838349 8310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:36:28.839946 8310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:28.840153 8310 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:28.840305 8310 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:28.842485 8315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:28.842509 8318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.842612 8310 server_base.cc:1060] running on GCE node
W20260319 16:36:28.842515 8316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:28.842864 8310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:28.843024 8310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:28.844143 8310 hybrid_clock.cc:648] HybridClock initialized: now 1773938188844130 us; error 25 us; skew 500 ppm
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:28.846401 8310 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:36:28.846899 8310 webserver.cc:492] Webserver started at http://127.24.187.195:39355/ using document root <none> and password file <none>
I20260319 16:36:28.847074 8310 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:28.847108 8310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:28.847211 8310 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:28.848157 8310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "1d1d6bd90ed04dee8e87343043731e47"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "733f84cfc19a2e279ea06df0779025b5"
server_key_iv: "19ce76d7d0fc8dac4aefdf5bd76b210e"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.848443 8310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "1d1d6bd90ed04dee8e87343043731e47"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "733f84cfc19a2e279ea06df0779025b5"
server_key_iv: "19ce76d7d0fc8dac4aefdf5bd76b210e"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.849561 8310 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:36:28.850273 8325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.850421 8310 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.850476 8310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "1d1d6bd90ed04dee8e87343043731e47"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "733f84cfc19a2e279ea06df0779025b5"
server_key_iv: "19ce76d7d0fc8dac4aefdf5bd76b210e"
server_key_version: "encryptionkey@0"
I20260319 16:36:28.850535 8310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:28.859563 8310 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:28.861304 8310 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:28.861424 8310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:28.861634 8310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:28.861989 8310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:28.862025 8310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.862053 8310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:28.862077 8310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:28.867477 8310 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:44335
I20260319 16:36:28.867530 8438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:44335 every 8 connection(s)
I20260319 16:36:28.867777 8310 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:36:28 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938188, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:28.875412 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.868348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:33515 (local address 127.24.187.254:39639)
0319 16:36:28.868398 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.868400 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.868527 (+ 127us) server_negotiation.cc:408] Connection header received
0319 16:36:28.868970 (+ 443us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.868971 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.868979 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.869041 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:28.869742 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.870220 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.870740 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.870872 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.872929 (+ 2057us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:28.872934 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:28.872934 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:28.872942 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:28.874229 (+ 1287us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.874616 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.874617 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.874617 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.874643 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.874789 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.874790 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.874790 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.874876 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:28.874930 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.875246 (+ 316us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.875346 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:36:28.875479 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.868303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:33515)
0319 16:36:28.868431 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.868460 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.868904 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.869086 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.869090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.869349 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:28.869647 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.869650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.870284 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.870285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.870653 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.870655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.870740 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.871133 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:28.871147 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:28.872850 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:28.874303 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:28.874308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:28.874310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:28.874517 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:28.874699 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:28.874699 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:28.874700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:28.874743 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:28.875003 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:28.875009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:28.875114 (+ 105us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.875209 (+ 95us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.875308 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260319 16:36:28.876019 8439 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39639
I20260319 16:36:28.876101 8439 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:28.876269 8439 heartbeater.cc:507] Master 127.24.187.254:39639 requested a full tablet report, sending...
I20260319 16:36:28.876649 7973 ts_manager.cc:194] Registered new tserver with Master: 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335)
I20260319 16:36:28.876760 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 8310
I20260319 16:36:28.876860 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:36:28.877039 25327 external_mini_cluster.cc:1468] Setting key 5915aee5ebb0040db48a47da5dba0f9f
I20260319 16:36:28.877137 7973 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:33515
I20260319 16:36:28.878014 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:36:28.883939 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.879254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.24.187.254:39639)
0319 16:36:28.879296 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.879297 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.879401 (+ 104us) server_negotiation.cc:408] Connection header received
0319 16:36:28.879440 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.879441 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.879449 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.879496 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:28.880056 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.880510 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.881045 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.881144 (+ 99us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.882053 (+ 909us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:28.882056 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:28.882057 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:28.882063 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:28.883223 (+ 1160us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.883470 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.883471 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.883471 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.883487 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:28.883589 (+ 102us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:28.883589 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:28.883589 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:28.883682 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:28.883749 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.883812 (+ 63us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.883866 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:36:28.885076 7973 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:36:28.886371 7973 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:36:28.890957 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.888324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:44335 (local address 127.0.0.1:59956)
0319 16:36:28.888615 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.888623 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.888657 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.888866 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.888869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.888870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:28.889088 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.889089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.890091 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.890092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.890718 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.890720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.890778 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.890815 (+ 37us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.890853 (+ 38us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.890891 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":46,"threads_started":1}
I20260319 16:36:28.891561 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.888322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43453 (local address 127.0.0.1:47966)
0319 16:36:28.888453 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.888481 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.888546 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.889167 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.889168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.889169 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:28.889324 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.889326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.890692 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.890693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.891363 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.891366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.891433 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.891437 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.891465 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.891500 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260319 16:36:28.891674 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.888332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42519 (local address 127.0.0.1:46428)
0319 16:36:28.888711 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.888721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.888774 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.889024 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.889025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.889027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:28.889189 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.889191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.890631 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.890632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.891484 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.891486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.891544 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.891548 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.891574 (+ 26us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.891612 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":75,"threads_started":1}
I20260319 16:36:28.892112 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.888538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.24.187.194:43453)
0319 16:36:28.888705 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.888712 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.888723 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:36:28.888761 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.888764 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.888780 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.888847 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:28.889410 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.890573 (+ 1163us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.891473 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.891975 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.892008 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.892033 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.892060 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260319 16:36:28.892143 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.888490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.24.187.195:44335)
0319 16:36:28.888743 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.888749 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.888755 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:36:28.888780 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.888781 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.888793 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.888847 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:28.889164 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.890012 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.891458 (+ 1446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.891954 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.892005 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.892032 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.892072 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":73,"threads_started":1}
I20260319 16:36:28.892519 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.888519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.24.187.193:42519)
0319 16:36:28.888886 (+ 367us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.888893 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.888902 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:36:28.888937 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.888939 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.888953 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.889006 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:28.889334 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.890548 (+ 1214us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.891843 (+ 1295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.892341 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.892381 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.892410 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.892448 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":72,"threads_started":1}
I20260319 16:36:28.892668 8237 tablet_service.cc:1511] Processing CreateTablet for tablet 35e8fcbbb7a640ea936276433afd1e43 (DEFAULT_TABLE table=test-table [id=8e9e9ce47e3d4b138ce264e02693b388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:28.892705 8373 tablet_service.cc:1511] Processing CreateTablet for tablet 35e8fcbbb7a640ea936276433afd1e43 (DEFAULT_TABLE table=test-table [id=8e9e9ce47e3d4b138ce264e02693b388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:28.892946 8237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e8fcbbb7a640ea936276433afd1e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:28.892953 8373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e8fcbbb7a640ea936276433afd1e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:28.892982 8101 tablet_service.cc:1511] Processing CreateTablet for tablet 35e8fcbbb7a640ea936276433afd1e43 (DEFAULT_TABLE table=test-table [id=8e9e9ce47e3d4b138ce264e02693b388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:28.893214 8101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35e8fcbbb7a640ea936276433afd1e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:28.895156 8456 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Bootstrap starting.
I20260319 16:36:28.895656 8458 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap starting.
I20260319 16:36:28.895702 8456 tablet_bootstrap.cc:654] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:28.895993 8456 log.cc:826] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:28.896219 8458 tablet_bootstrap.cc:654] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:28.896456 8458 log.cc:826] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:28.896706 8456 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: No bootstrap required, opened a new log
I20260319 16:36:28.896759 8457 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Bootstrap starting.
I20260319 16:36:28.896770 8456 ts_tablet_manager.cc:1403] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:28.897037 8458 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: No bootstrap required, opened a new log
I20260319 16:36:28.897092 8458 ts_tablet_manager.cc:1403] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:28.897272 8457 tablet_bootstrap.cc:654] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:28.897557 8457 log.cc:826] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:28.898233 8457 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: No bootstrap required, opened a new log
I20260319 16:36:28.898303 8457 ts_tablet_manager.cc:1403] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260319 16:36:28.898357 8456 raft_consensus.cc:359] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.898486 8456 raft_consensus.cc:385] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:28.898514 8456 raft_consensus.cc:740] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbea135349c1477c9fd86ec60a7b9f75, State: Initialized, Role: FOLLOWER
I20260319 16:36:28.898659 8458 raft_consensus.cc:359] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.898797 8458 raft_consensus.cc:385] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:28.898823 8458 raft_consensus.cc:740] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1cfc6aad0e44b4bd8da5cfa9818d38, State: Initialized, Role: FOLLOWER
I20260319 16:36:28.898622 8456 consensus_queue.cc:260] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.898947 8458 consensus_queue.cc:260] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.899067 8303 heartbeater.cc:499] Master 127.24.187.254:39639 was elected leader, sending a full tablet report...
I20260319 16:36:28.899050 8456 ts_tablet_manager.cc:1434] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260319 16:36:28.899287 8458 ts_tablet_manager.cc:1434] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:36:28.899317 8167 heartbeater.cc:499] Master 127.24.187.254:39639 was elected leader, sending a full tablet report...
I20260319 16:36:28.899925 8457 raft_consensus.cc:359] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.900058 8457 raft_consensus.cc:385] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:28.900125 8457 raft_consensus.cc:740] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1d6bd90ed04dee8e87343043731e47, State: Initialized, Role: FOLLOWER
I20260319 16:36:28.900236 8457 consensus_queue.cc:260] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.900457 8457 ts_tablet_manager.cc:1434] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:36:28.900499 8439 heartbeater.cc:499] Master 127.24.187.254:39639 was elected leader, sending a full tablet report...
W20260319 16:36:28.963586 8304 tablet.cc:2379] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:36:28.978940 8462 raft_consensus.cc:493] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:36:28.979069 8462 raft_consensus.cc:515] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.979452 8462 leader_election.cc:290] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335), 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519)
I20260319 16:36:28.981547 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.979650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42519 (local address 127.24.187.194:53153)
0319 16:36:28.979710 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.979721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.979767 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.979938 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.979940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.979942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:28.980125 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.980126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.980751 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.980752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.981339 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.981341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.981413 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.981418 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.981445 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.981483 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260319 16:36:28.982041 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.979679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:53153 (local address 127.24.187.193:42519)
0319 16:36:28.979721 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.979723 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.979755 (+ 32us) server_negotiation.cc:408] Connection header received
0319 16:36:28.979855 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.979857 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.979866 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.979924 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:28.980208 (+ 284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.980682 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.981516 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.981948 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.981956 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.981974 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.981995 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:36:28.982138 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.979653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:44335 (local address 127.24.187.194:56741)
0319 16:36:28.979832 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:28.979841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:28.979890 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:28.980001 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:28.980001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:28.980003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:28.980219 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.980220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.981109 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.981110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:28.981927 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:28.981930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.981992 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.981997 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:28.982023 (+ 26us) client_negotiation.cc:240] Negotiation successful
0319 16:36:28.982055 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":9,"thread_start_us":110,"threads_started":1}
I20260319 16:36:28.982367 8121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35e8fcbbb7a640ea936276433afd1e43" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" is_pre_election: true
I20260319 16:36:28.982523 8121 raft_consensus.cc:2468] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbea135349c1477c9fd86ec60a7b9f75 in term 0.
I20260319 16:36:28.982656 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:28.979674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:56741 (local address 127.24.187.195:44335)
0319 16:36:28.979720 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:28.979722 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:28.979901 (+ 179us) server_negotiation.cc:408] Connection header received
0319 16:36:28.979935 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:28.979937 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:28.979946 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:28.979991 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:28.980284 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.981007 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:28.982009 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:28.982526 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:28.982535 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:28.982554 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:36:28.982583 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:36:28.982746 8192 leader_election.cc:304] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1cfc6aad0e44b4bd8da5cfa9818d38, cbea135349c1477c9fd86ec60a7b9f75; no voters:
I20260319 16:36:28.982864 8462 raft_consensus.cc:2804] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:36:28.982931 8462 raft_consensus.cc:493] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:36:28.982949 8462 raft_consensus.cc:3060] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:28.983764 8462 raft_consensus.cc:515] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.983906 8462 leader_election.cc:290] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 election: Requested vote from peers 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335), 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519)
I20260319 16:36:28.984081 8392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35e8fcbbb7a640ea936276433afd1e43" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d6bd90ed04dee8e87343043731e47"
I20260319 16:36:28.984061 8393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35e8fcbbb7a640ea936276433afd1e43" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d6bd90ed04dee8e87343043731e47" is_pre_election: true
I20260319 16:36:28.984194 8392 raft_consensus.cc:3060] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:28.984907 8392 raft_consensus.cc:2468] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbea135349c1477c9fd86ec60a7b9f75 in term 1.
I20260319 16:36:28.985014 8121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35e8fcbbb7a640ea936276433afd1e43" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38"
I20260319 16:36:28.985093 8121 raft_consensus.cc:3060] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:28.985116 8191 leader_election.cc:304] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1d6bd90ed04dee8e87343043731e47, cbea135349c1477c9fd86ec60a7b9f75; no voters:
I20260319 16:36:28.985236 8462 raft_consensus.cc:2804] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:28.985412 8462 raft_consensus.cc:697] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 LEADER]: Becoming Leader. State: Replica: cbea135349c1477c9fd86ec60a7b9f75, State: Running, Role: LEADER
I20260319 16:36:28.985505 8462 consensus_queue.cc:237] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:28.985954 8121 raft_consensus.cc:2468] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbea135349c1477c9fd86ec60a7b9f75 in term 1.
I20260319 16:36:28.986286 7972 catalog_manager.cc:5671] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbea135349c1477c9fd86ec60a7b9f75 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "cbea135349c1477c9fd86ec60a7b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } health_report { overall_health: UNKNOWN } } }
I20260319 16:36:29.008971 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.007018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.24.187.194:43453)
0319 16:36:29.007076 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.007078 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.007085 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:36:29.007125 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.007126 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.007140 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.007189 (+ 49us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:36:29.007459 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.007920 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.008475 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.008612 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.008662 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:36:29.008747 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:36:29.008812 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.008872 (+ 60us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.008896 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260319 16:36:29.010500 8392 raft_consensus.cc:1275] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Refusing update from remote peer cbea135349c1477c9fd86ec60a7b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:29.010532 8121 raft_consensus.cc:1275] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Refusing update from remote peer cbea135349c1477c9fd86ec60a7b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:29.010772 8466 consensus_queue.cc:1048] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:29.010890 8462 consensus_queue.cc:1048] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:29.013159 8469 mvcc.cc:204] Tried to move back new op lower bound from 7266050822184689664 to 7266050822085120000. Current Snapshot: MvccSnapshot[applied={T|T < 7266050822184689664}]
I20260319 16:36:29.014074 8471 mvcc.cc:204] Tried to move back new op lower bound from 7266050822184689664 to 7266050822085120000. Current Snapshot: MvccSnapshot[applied={T|T < 7266050822184689664 or (T in {7266050822184689664})}]
I20260319 16:36:29.015202 7973 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:29.015302 7973 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:29.016023 7973 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Sending DeleteTablet for 3 replicas of tablet 35e8fcbbb7a640ea936276433afd1e43
I20260319 16:36:29.016306 8237 tablet_service.cc:1558] Processing DeleteTablet for tablet 35e8fcbbb7a640ea936276433afd1e43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:29 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:47966
I20260319 16:36:29.016319 8373 tablet_service.cc:1558] Processing DeleteTablet for tablet 35e8fcbbb7a640ea936276433afd1e43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:29 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:59956
I20260319 16:36:29.016371 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 8037
I20260319 16:36:29.016986 8479 tablet_replica.cc:333] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: stopping tablet replica
I20260319 16:36:29.017066 8479 raft_consensus.cc:2243] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:36:29.017150 8479 raft_consensus.cc:2272] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:29.018179 8479 ts_tablet_manager.cc:1916] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:29.019877 8479 ts_tablet_manager.cc:1929] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:29.019981 8479 log.cc:1199] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/35e8fcbbb7a640ea936276433afd1e43
I20260319 16:36:29.020323 8479 ts_tablet_manager.cc:1950] T 35e8fcbbb7a640ea936276433afd1e43 P 1d1d6bd90ed04dee8e87343043731e47: Deleting consensus metadata
I20260319 16:36:29.020761 7958 catalog_manager.cc:5002] TS 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335): tablet 35e8fcbbb7a640ea936276433afd1e43 (table test-table [id=8e9e9ce47e3d4b138ce264e02693b388]) successfully deleted
I20260319 16:36:29.021884 8480 tablet_replica.cc:333] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: stopping tablet replica
I20260319 16:36:29.022028 8480 raft_consensus.cc:2243] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:36:29.022136 8480 raft_consensus.cc:2272] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:29.022584 8480 ts_tablet_manager.cc:1916] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260319 16:36:29.023046 7959 connection.cc:537] client connection to 127.24.187.193:42519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260319 16:36:29.023130 7959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260319 16:36:29.023345 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 8174
I20260319 16:36:29.023725 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.023611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42519 (local address 127.0.0.1:46434)
0319 16:36:29.023660 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.023690 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260319 16:36:29.023986 7959 catalog_manager.cc:4729] TS 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35e8fcbbb7a640ea936276433afd1e43: Network error: Client connection negotiation failed: client connection to 127.24.187.193:42519: connect: Connection refused (error 111)
I20260319 16:36:29.029067 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.028930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43453 (local address 127.0.0.1:47988)
0319 16:36:29.028999 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.029034 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.194:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260319 16:36:29.029143 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 8310
I20260319 16:36:29.033411 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 7941
I20260319 16:36:29.038174 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39639
--webserver_interface=127.24.187.254
--webserver_port=33401
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:29.112681 8482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:29.112854 8482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:29.112877 8482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:29.114421 8482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:29.114470 8482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:29.114485 8482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:29.114498 8482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:29.114518 8482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:29.116161 8482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=33401
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:29.116365 8482 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:29.116550 8482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:29.123704 8488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.123754 8482 server_base.cc:1060] running on GCE node
W20260319 16:36:29.123698 8489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:29.123698 8491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.124133 8482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:29.124365 8482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:29.125509 8482 hybrid_clock.cc:648] HybridClock initialized: now 1773938189125492 us; error 33 us; skew 500 ppm
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:29.127866 8482 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:29.128364 8482 webserver.cc:492] Webserver started at http://127.24.187.254:33401/ using document root <none> and password file <none>
I20260319 16:36:29.128566 8482 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:29.128619 8482 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:29.129930 8482 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:29.130532 8498 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.130730 8482 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.130789 8482 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "f8e773f5586ac879160192c194c0e75a"
server_key_iv: "496d381593d3556da4b7a70b47f5a891"
server_key_version: "encryptionkey@0"
I20260319 16:36:29.131094 8482 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:29.148319 8482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:29.150225 8482 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:29.150369 8482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:29.154460 8482 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39639
I20260319 16:36:29.154495 8550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39639 every 8 connection(s)
I20260319 16:36:29.154846 8482 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:29.155977 8551 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:36:29.156541 8551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Bootstrap starting.
I20260319 16:36:29.162757 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 8482
I20260319 16:36:29.163048 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:42519
--local_ip_for_outbound_sockets=127.24.187.193
--tserver_master_addrs=127.24.187.254:39639
--webserver_port=39657
--webserver_interface=127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:29.165064 8551 log.cc:826] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:29.167212 8551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:36:29.167490 8551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Bootstrap complete.
I20260319 16:36:29.169431 8551 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.169644 8551 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa3a6209d3f4b1396b7f1c726b2e3b6, State: Initialized, Role: FOLLOWER
I20260319 16:36:29.169800 8551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.169878 8551 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:36:29.169934 8551 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:36:29.169984 8551 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 1 FOLLOWER]: Advancing to term 2
I20260319 16:36:29.170874 8551 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.170987 8551 leader_election.cc:304] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa3a6209d3f4b1396b7f1c726b2e3b6; no voters:
I20260319 16:36:29.171149 8551 leader_election.cc:290] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260319 16:36:29.171202 8555 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260319 16:36:29.171360 8555 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 LEADER]: Becoming Leader. State: Replica: 5fa3a6209d3f4b1396b7f1c726b2e3b6, State: Running, Role: LEADER
I20260319 16:36:29.171404 8551 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:29.171447 8555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.171834 8557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa3a6209d3f4b1396b7f1c726b2e3b6. Latest consensus state: current_term: 2 leader_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } } }
I20260319 16:36:29.171922 8557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:29.172048 8556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } } }
I20260319 16:36:29.172101 8556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:29.172231 8559 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:29.172991 8559 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:29.173108 8559 catalog_manager.cc:1269] Loaded cluster ID: be663a29f6d949d986c9f02850e0c161
I20260319 16:36:29.173174 8559 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:36:29.173431 8559 catalog_manager.cc:1324] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260319 16:36:29.175287 8559 catalog_manager.cc:1334] IPKI private key successfully encrypted.
I20260319 16:36:29.178721 8559 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:36:29.178900 8559 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Loaded TSK: 0
I20260319 16:36:29.179065 8559 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260319 16:36:29.257383 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:29.257586 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:29.257622 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:29.259217 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:29.259269 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:29.259295 8553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:36:29.261021 8553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:42519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=39657
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:29.261224 8553 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:29.261404 8553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:29.263845 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:29.263847 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:29.263847 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.264137 8553 server_base.cc:1060] running on GCE node
I20260319 16:36:29.264276 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:29.264498 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:29.265645 8553 hybrid_clock.cc:648] HybridClock initialized: now 1773938189265626 us; error 30 us; skew 500 ppm
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:29.268013 8553 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:36:29.268560 8553 webserver.cc:492] Webserver started at http://127.24.187.193:39657/ using document root <none> and password file <none>
I20260319 16:36:29.268747 8553 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:29.268793 8553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:29.270017 8553 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:36:29.270665 8589 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.270823 8553 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:36:29.270881 8553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "3a974ef1544f2ff83744cd577d28dc3f"
server_key_iv: "136cfc55e299b726b3b971b391dc6ad8"
server_key_version: "encryptionkey@0"
I20260319 16:36:29.271157 8553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:29.283492 8553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:29.285214 8553 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:29.285346 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:29.285550 8553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:29.286010 8596 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260319 16:36:29.287067 8553 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260319 16:36:29.287113 8553 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:29.287149 8553 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260319 16:36:29.287659 8553 ts_tablet_manager.cc:616] Registered 1 tablets
I20260319 16:36:29.287689 8553 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:29.287770 8596 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap starting.
I20260319 16:36:29.294786 8553 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:42519
I20260319 16:36:29.295128 8553 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:36:29.295608 8703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:42519 every 8 connection(s)
I20260319 16:36:29.298121 8596 log.cc:826] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:29.298357 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 8553
I20260319 16:36:29.298650 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:43453
--local_ip_for_outbound_sockets=127.24.187.194
--tserver_master_addrs=127.24.187.254:39639
--webserver_port=39543
--webserver_interface=127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:29.301111 8596 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:36:29.301434 8596 tablet_bootstrap.cc:492] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap complete.
I20260319 16:36:29.302536 8596 ts_tablet_manager.cc:1403] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260319 16:36:29.303709 8596 raft_consensus.cc:359] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.303992 8596 raft_consensus.cc:740] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1cfc6aad0e44b4bd8da5cfa9818d38, State: Initialized, Role: FOLLOWER
I20260319 16:36:29.304090 8596 consensus_queue.cc:260] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.304340 8596 ts_tablet_manager.cc:1434] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:29.308972 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.295875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:54819 (local address 127.24.187.254:39639)
0319 16:36:29.299578 (+ 3703us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.299589 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.299617 (+ 28us) server_negotiation.cc:408] Connection header received
0319 16:36:29.299732 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.299739 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.299786 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.301035 (+ 1249us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:29.301144 (+ 109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.302431 (+ 1287us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.304475 (+ 2044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.304660 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.305767 (+ 1107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:29.305798 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:29.305798 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:29.305816 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:29.307468 (+ 1652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.307846 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.307849 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.307849 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.307893 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.308153 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.308154 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.308155 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.308333 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:29.308423 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.308690 (+ 267us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.308800 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3651,"thread_start_us":70,"threads_started":1}
I20260319 16:36:29.308986 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.295805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:54819)
0319 16:36:29.298407 (+ 2602us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.298441 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.299070 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.300040 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.300046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.300475 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:29.300902 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.300907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.302539 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.302540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.303073 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.303076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.303204 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.303776 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:29.303798 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:29.305648 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:29.307571 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:29.307577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:29.307579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:29.307773 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:29.308028 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:29.308029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:29.308030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:29.308097 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:29.308403 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:29.308407 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:29.308517 (+ 110us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.308626 (+ 109us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.308785 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2562,"thread_start_us":80,"threads_started":1}
I20260319 16:36:29.309625 8704 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39639
I20260319 16:36:29.309731 8704 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:29.309913 8704 heartbeater.cc:507] Master 127.24.187.254:39639 requested a full tablet report, sending...
I20260319 16:36:29.310506 8515 ts_manager.cc:194] Registered new tserver with Master: 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519)
I20260319 16:36:29.311496 8515 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:54819
I20260319 16:36:29.312175 8704 heartbeater.cc:499] Master 127.24.187.254:39639 was elected leader, sending a full tablet report...
I20260319 16:36:29.319950 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.310995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42519 (local address 127.0.0.1:46436)
0319 16:36:29.313832 (+ 2837us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.313862 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.313927 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.317994 (+ 4067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.317997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.317999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:29.318223 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.318225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.319083 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.319084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.319686 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.319688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.319752 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.319783 (+ 31us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.319826 (+ 43us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.319864 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2807,"thread_start_us":94,"threads_started":1}
I20260319 16:36:29.320477 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.311209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.24.187.193:42519)
0319 16:36:29.317845 (+ 6636us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.317851 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.317861 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:36:29.317889 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.317890 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.317904 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.317974 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:29.318292 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.319008 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.319759 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.320332 (+ 573us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.320370 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.320393 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.320426 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6605,"thread_start_us":69,"threads_started":1}
I20260319 16:36:29.320729 8638 tablet_service.cc:1558] Processing DeleteTablet for tablet 35e8fcbbb7a640ea936276433afd1e43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:29 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:46436
I20260319 16:36:29.321177 8713 tablet_replica.cc:333] stopping tablet replica
I20260319 16:36:29.321257 8713 raft_consensus.cc:2243] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:36:29.321321 8713 raft_consensus.cc:2272] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:29.321599 8713 ts_tablet_manager.cc:1916] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:29.323412 8713 ts_tablet_manager.cc:1929] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:29.323474 8713 log.cc:1199] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/35e8fcbbb7a640ea936276433afd1e43
I20260319 16:36:29.323699 8713 ts_tablet_manager.cc:1950] T 35e8fcbbb7a640ea936276433afd1e43 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Deleting consensus metadata
I20260319 16:36:29.324141 8501 catalog_manager.cc:5002] TS 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519): tablet 35e8fcbbb7a640ea936276433afd1e43 (table test-table [id=8e9e9ce47e3d4b138ce264e02693b388]) successfully deleted
W20260319 16:36:29.397497 8709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:29.397673 8709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:29.397694 8709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:29.399199 8709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:29.399243 8709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:29.399267 8709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:36:29.400869 8709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:43453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=39543
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:29.401057 8709 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:29.401222 8709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:29.403570 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:29.403577 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.403682 8709 server_base.cc:1060] running on GCE node
W20260319 16:36:29.403580 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.403930 8709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:29.404114 8709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:29.405246 8709 hybrid_clock.cc:648] HybridClock initialized: now 1773938189405227 us; error 28 us; skew 500 ppm
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:29.407529 8709 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:36:29.407968 8709 webserver.cc:492] Webserver started at http://127.24.187.194:39543/ using document root <none> and password file <none>
I20260319 16:36:29.408128 8709 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:29.408176 8709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:29.409310 8709 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:36:29.410027 8728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.410229 8709 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:29.410290 8709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "cbea135349c1477c9fd86ec60a7b9f75"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "2387295af3a0aad324f22b53baf4c7ef"
server_key_iv: "476dcf620ec9b63ae1846ef581c9ade2"
server_key_version: "encryptionkey@0"
I20260319 16:36:29.410591 8709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:29.436419 8709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:29.438467 8709 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:29.438599 8709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:29.438831 8709 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:29.439239 8735 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260319 16:36:29.439989 8735 ts_tablet_manager.cc:1863] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260319 16:36:29.440235 8735 ts_tablet_manager.cc:1916] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:29.442170 8735 ts_tablet_manager.cc:1929] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:29.442255 8735 log.cc:1199] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/35e8fcbbb7a640ea936276433afd1e43
I20260319 16:36:29.442510 8735 ts_tablet_manager.cc:1950] T 35e8fcbbb7a640ea936276433afd1e43 P cbea135349c1477c9fd86ec60a7b9f75: Deleting consensus metadata
I20260319 16:36:29.442706 8709 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260319 16:36:29.442768 8709 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260319 16:36:29.442804 8709 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:29.442821 8709 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.448231 8709 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:43453
I20260319 16:36:29.448273 8842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:43453 every 8 connection(s)
I20260319 16:36:29.448520 8709 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:29.456296 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.449134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:40311 (local address 127.24.187.254:39639)
0319 16:36:29.449209 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.449212 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.449311 (+ 99us) server_negotiation.cc:408] Connection header received
0319 16:36:29.450119 (+ 808us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.450121 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.450133 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.450197 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:29.450924 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.451439 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.452052 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.452185 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.454150 (+ 1965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:29.454154 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:29.454154 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:29.454162 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:29.455385 (+ 1223us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.455729 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.455730 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.455730 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.455748 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.455869 (+ 121us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.455870 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.455870 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.455951 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:29.455991 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.456168 (+ 177us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.456232 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260319 16:36:29.456434 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.449093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:40311)
0319 16:36:29.449218 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.449245 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.450003 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.450250 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.450253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.450574 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:29.450822 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.450825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.451552 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.451553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.451962 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.451964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.452080 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.452635 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:29.452651 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:29.454043 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:29.455466 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:29.455471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:29.455473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:29.455666 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:29.455786 (+ 120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:29.455786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:29.455787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:29.455833 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:29.455987 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:29.455989 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:29.456082 (+ 93us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.456155 (+ 73us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.456247 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260319 16:36:29.456898 8843 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39639
I20260319 16:36:29.456975 8843 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:29.457139 8843 heartbeater.cc:507] Master 127.24.187.254:39639 requested a full tablet report, sending...
I20260319 16:36:29.457522 8515 ts_manager.cc:194] Registered new tserver with Master: cbea135349c1477c9fd86ec60a7b9f75 (127.24.187.194:43453)
I20260319 16:36:29.457569 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 8709
I20260319 16:36:29.457857 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:44335
--local_ip_for_outbound_sockets=127.24.187.195
--tserver_master_addrs=127.24.187.254:39639
--webserver_port=39355
--webserver_interface=127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:29.457938 8515 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:40311
W20260319 16:36:29.531908 8846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:29.532068 8846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:29.532090 8846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:29.533551 8846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:29.533592 8846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:29.533617 8846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:36:29.535260 8846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:44335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=39355
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:29.535450 8846 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:29.535602 8846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:29.537899 8851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:29.537901 8852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:29.537899 8854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.538079 8846 server_base.cc:1060] running on GCE node
I20260319 16:36:29.538249 8846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:29.538443 8846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:29.539590 8846 hybrid_clock.cc:648] HybridClock initialized: now 1773938189539571 us; error 38 us; skew 500 ppm
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:29.541953 8846 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:36:29.542433 8846 webserver.cc:492] Webserver started at http://127.24.187.195:39355/ using document root <none> and password file <none>
I20260319 16:36:29.542606 8846 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:29.542659 8846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:29.543871 8846 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:29.544530 8861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.544678 8846 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:36:29.544765 8846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "1d1d6bd90ed04dee8e87343043731e47"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "733f84cfc19a2e279ea06df0779025b5"
server_key_iv: "19ce76d7d0fc8dac4aefdf5bd76b210e"
server_key_version: "encryptionkey@0"
I20260319 16:36:29.545084 8846 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:29.565464 8846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:29.567237 8846 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:29.567341 8846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:29.567561 8846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:29.567857 8846 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:29.567886 8846 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.567907 8846 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:29.567930 8846 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.573127 8846 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:44335
I20260319 16:36:29.573215 8975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:44335 every 8 connection(s)
I20260319 16:36:29.573549 8846 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:29.581552 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.574179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:37903 (local address 127.24.187.254:39639)
0319 16:36:29.574220 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.574222 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.574423 (+ 201us) server_negotiation.cc:408] Connection header received
0319 16:36:29.574911 (+ 488us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.574913 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.574927 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.574975 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:29.575764 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.576239 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.576776 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.576891 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.579225 (+ 2334us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:29.579229 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:29.579229 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:29.579237 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:29.580394 (+ 1157us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.580747 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.580749 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.580749 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.580776 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.580963 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.580965 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.580965 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.581061 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:29.581129 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.581407 (+ 278us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.581485 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260319 16:36:29.581645 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.574158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:37903)
0319 16:36:29.574303 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.574342 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.574801 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.575067 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.575071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.575385 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:29.575665 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.575668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.576347 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.576348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.576715 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.576717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.576805 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.577226 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:29.577241 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:29.579133 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:29.580467 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:29.580472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:29.580474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:29.580663 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:29.580834 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:29.580834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:29.580835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:29.580884 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:29.581159 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:29.581164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:29.581276 (+ 112us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.581368 (+ 92us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.581488 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260319 16:36:29.582126 8976 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39639
I20260319 16:36:29.582207 8976 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:29.582399 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 8846
I20260319 16:36:29.582494 8976 heartbeater.cc:507] Master 127.24.187.254:39639 requested a full tablet report, sending...
I20260319 16:36:29.582918 8515 ts_manager.cc:194] Registered new tserver with Master: 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335)
I20260319 16:36:29.583312 8515 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:37903
I20260319 16:36:29.587666 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.582804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.24.187.254:39639)
0319 16:36:29.582847 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.582848 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.582942 (+ 94us) server_negotiation.cc:408] Connection header received
0319 16:36:29.583016 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.583017 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.583026 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.583081 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:29.583783 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.584226 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.584761 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.584917 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.585745 (+ 828us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:29.585748 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:29.585748 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:29.585755 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:29.586755 (+ 1000us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.587038 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.587039 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.587040 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.587059 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:29.587181 (+ 122us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:29.587181 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:29.587182 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:29.587255 (+ 73us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:29.587391 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.587494 (+ 103us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.587569 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:36:29.588301 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:36:29.590384 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.588579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.24.187.254:39639)
0319 16:36:29.588618 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.588619 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.588701 (+ 82us) server_negotiation.cc:408] Connection header received
0319 16:36:29.588763 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.588764 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.588781 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.588835 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:36:29.589030 (+ 195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.589481 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.589997 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.590098 (+ 101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.590129 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:36:29.590199 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:36:29.590253 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.590295 (+ 42us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.590311 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:36:29.590583 8515 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:36:29.591811 8515 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:36:29.593655 8638 tablet_service.cc:1511] Processing CreateTablet for tablet 4aa24b98d69a4b519947558a52e318b5 (DEFAULT_TABLE table=test-table [id=2c306ff81c254d0480afd265eef12ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:29.593859 8638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa24b98d69a4b519947558a52e318b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:29.595105 8596 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap starting.
I20260319 16:36:29.595460 8596 tablet_bootstrap.cc:654] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:29.596040 8596 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: No bootstrap required, opened a new log
I20260319 16:36:29.596094 8596 ts_tablet_manager.cc:1403] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:29.596195 8596 raft_consensus.cc:359] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.596259 8596 raft_consensus.cc:385] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:29.596302 8596 raft_consensus.cc:740] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1cfc6aad0e44b4bd8da5cfa9818d38, State: Initialized, Role: FOLLOWER
I20260319 16:36:29.596347 8596 consensus_queue.cc:260] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.596452 8596 ts_tablet_manager.cc:1434] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.596477 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.593435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43453 (local address 127.0.0.1:48000)
0319 16:36:29.593485 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.593495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.593530 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.594018 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.594019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.594021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:29.594182 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.594184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.595491 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.595492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.596261 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.596264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.596332 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.596337 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.596366 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.596409 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260319 16:36:29.597137 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.593661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.24.187.195:44335)
0319 16:36:29.593899 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.593905 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.593913 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:36:29.593944 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.593945 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.593958 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.594023 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:29.594286 (+ 263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.595414 (+ 1128us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.596471 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.596959 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.597009 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.597036 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.597066 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260319 16:36:29.597239 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.593532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.24.187.194:43453)
0319 16:36:29.593842 (+ 310us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.593849 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.593859 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:36:29.593898 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.593900 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.593918 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.594242 (+ 324us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:29.594324 (+ 82us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.595426 (+ 1102us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.596579 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.597084 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.597130 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.597159 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.597191 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260319 16:36:29.597347 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.593483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:44335 (local address 127.0.0.1:59958)
0319 16:36:29.593654 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.593663 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.593731 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.594029 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.594030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.594032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:29.594210 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.594212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.595510 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.595511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.596237 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.596240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.596307 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.596312 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.596337 (+ 25us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.596385 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":54,"threads_started":1}
I20260319 16:36:29.597654 8777 tablet_service.cc:1511] Processing CreateTablet for tablet 4aa24b98d69a4b519947558a52e318b5 (DEFAULT_TABLE table=test-table [id=2c306ff81c254d0480afd265eef12ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:29.597898 8777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa24b98d69a4b519947558a52e318b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:29.597924 8909 tablet_service.cc:1511] Processing CreateTablet for tablet 4aa24b98d69a4b519947558a52e318b5 (DEFAULT_TABLE table=test-table [id=2c306ff81c254d0480afd265eef12ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:29.598166 8909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa24b98d69a4b519947558a52e318b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:29.599238 8735 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Bootstrap starting.
I20260319 16:36:29.599725 8735 tablet_bootstrap.cc:654] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:29.599990 8735 log.cc:826] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:29.600425 8986 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Bootstrap starting.
I20260319 16:36:29.600641 8735 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: No bootstrap required, opened a new log
I20260319 16:36:29.600687 8735 ts_tablet_manager.cc:1403] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:29.600934 8986 tablet_bootstrap.cc:654] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:29.601156 8986 log.cc:826] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:29.601887 8986 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: No bootstrap required, opened a new log
I20260319 16:36:29.601949 8986 ts_tablet_manager.cc:1403] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:29.601931 8735 raft_consensus.cc:359] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.602056 8735 raft_consensus.cc:385] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:29.602079 8735 raft_consensus.cc:740] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbea135349c1477c9fd86ec60a7b9f75, State: Initialized, Role: FOLLOWER
I20260319 16:36:29.602172 8735 consensus_queue.cc:260] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.602586 8735 ts_tablet_manager.cc:1434] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:36:29.602628 8843 heartbeater.cc:499] Master 127.24.187.254:39639 was elected leader, sending a full tablet report...
I20260319 16:36:29.603713 8986 raft_consensus.cc:359] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.603816 8986 raft_consensus.cc:385] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:29.603833 8986 raft_consensus.cc:740] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1d6bd90ed04dee8e87343043731e47, State: Initialized, Role: FOLLOWER
I20260319 16:36:29.603893 8986 consensus_queue.cc:260] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.604077 8986 ts_tablet_manager.cc:1434] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:36:29.604133 8976 heartbeater.cc:499] Master 127.24.187.254:39639 was elected leader, sending a full tablet report...
W20260319 16:36:29.699034 8844 tablet.cc:2379] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:36:29.774731 8988 raft_consensus.cc:493] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:36:29.774865 8988 raft_consensus.cc:515] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.775228 8988 leader_election.cc:290] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335), 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519)
I20260319 16:36:29.777988 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.775388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:44335 (local address 127.24.187.194:45503)
0319 16:36:29.775453 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.775467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.775522 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.775809 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.775810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.775812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:29.776049 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.776051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.776896 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.776897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.777741 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.777744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.777818 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.777823 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.777863 (+ 40us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.777902 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260319 16:36:29.778124 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.775410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42519 (local address 127.24.187.194:55645)
0319 16:36:29.775675 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.775684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:29.775770 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:29.776065 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:29.776066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:29.776068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:29.776273 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.776274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.777069 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.777070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:29.777940 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:29.777942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.777999 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.778001 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:36:29.778028 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:36:29.778061 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":186,"threads_started":1}
I20260319 16:36:29.778601 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.775574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:55645 (local address 127.24.187.193:42519)
0319 16:36:29.775658 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.775660 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.775739 (+ 79us) server_negotiation.cc:408] Connection header received
0319 16:36:29.775848 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.775849 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.775937 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.775987 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:29.776344 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.776996 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.778017 (+ 1021us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.778473 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.778483 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.778506 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.778538 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260319 16:36:29.778690 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.775592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:45503 (local address 127.24.187.195:44335)
0319 16:36:29.775689 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.775691 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.775699 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:36:29.775726 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.775727 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.775736 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.775788 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:29.776125 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.776789 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.777823 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.778569 (+ 746us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.778577 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.778599 (+ 22us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.778633 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:36:29.778923 8929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4aa24b98d69a4b519947558a52e318b5" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d6bd90ed04dee8e87343043731e47" is_pre_election: true
I20260319 16:36:29.778898 8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4aa24b98d69a4b519947558a52e318b5" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" is_pre_election: true
I20260319 16:36:29.779049 8929 raft_consensus.cc:2468] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbea135349c1477c9fd86ec60a7b9f75 in term 0.
I20260319 16:36:29.779057 8658 raft_consensus.cc:2468] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbea135349c1477c9fd86ec60a7b9f75 in term 0.
I20260319 16:36:29.779276 8730 leader_election.cc:304] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1d6bd90ed04dee8e87343043731e47, cbea135349c1477c9fd86ec60a7b9f75; no voters:
I20260319 16:36:29.779446 8988 raft_consensus.cc:2804] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:36:29.779508 8988 raft_consensus.cc:493] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:36:29.779525 8988 raft_consensus.cc:3060] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:29.780174 8988 raft_consensus.cc:515] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.780303 8988 leader_election.cc:290] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 election: Requested vote from peers 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335), 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519)
I20260319 16:36:29.780472 8658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4aa24b98d69a4b519947558a52e318b5" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38"
I20260319 16:36:29.780498 8929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4aa24b98d69a4b519947558a52e318b5" candidate_uuid: "cbea135349c1477c9fd86ec60a7b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1d6bd90ed04dee8e87343043731e47"
I20260319 16:36:29.780567 8929 raft_consensus.cc:3060] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:29.780565 8658 raft_consensus.cc:3060] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:29.781193 8929 raft_consensus.cc:2468] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbea135349c1477c9fd86ec60a7b9f75 in term 1.
I20260319 16:36:29.781213 8658 raft_consensus.cc:2468] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbea135349c1477c9fd86ec60a7b9f75 in term 1.
I20260319 16:36:29.781378 8730 leader_election.cc:304] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1d6bd90ed04dee8e87343043731e47, cbea135349c1477c9fd86ec60a7b9f75; no voters:
I20260319 16:36:29.781492 8988 raft_consensus.cc:2804] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:29.781638 8988 raft_consensus.cc:697] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 LEADER]: Becoming Leader. State: Replica: cbea135349c1477c9fd86ec60a7b9f75, State: Running, Role: LEADER
I20260319 16:36:29.781777 8988 consensus_queue.cc:237] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:29.782748 8515 catalog_manager.cc:5671] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbea135349c1477c9fd86ec60a7b9f75 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "cbea135349c1477c9fd86ec60a7b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } health_report { overall_health: UNKNOWN } } }
W20260319 16:36:29.795878 8705 tablet.cc:2379] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:36:29.824040 8977 tablet.cc:2379] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:36:29.825187 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.823041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.24.187.194:43453)
0319 16:36:29.823094 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:29.823096 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:29.823104 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:36:29.823147 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:29.823148 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:29.823163 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:29.823214 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:36:29.823543 (+ 329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.824036 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:29.824696 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:29.824832 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:29.824871 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:36:29.824956 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:36:29.825015 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:29.825095 (+ 80us) server_negotiation.cc:300] Negotiation successful
0319 16:36:29.825116 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260319 16:36:29.826804 8658 raft_consensus.cc:1275] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Refusing update from remote peer cbea135349c1477c9fd86ec60a7b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:29.826838 8929 raft_consensus.cc:1275] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Refusing update from remote peer cbea135349c1477c9fd86ec60a7b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:29.827150 8988 consensus_queue.cc:1048] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:29.827270 8991 consensus_queue.cc:1048] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:29.829296 8994 mvcc.cc:204] Tried to move back new op lower bound from 7266050825528061952 to 7266050825346965504. Current Snapshot: MvccSnapshot[applied={T|T < 7266050825528061952}]
I20260319 16:36:29.829567 8995 mvcc.cc:204] Tried to move back new op lower bound from 7266050825528061952 to 7266050825346965504. Current Snapshot: MvccSnapshot[applied={T|T < 7266050825528061952 or (T in {7266050825528061952})}]
I20260319 16:36:29.831429 8514 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:29.831528 8514 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:29.832254 8514 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Sending DeleteTablet for 3 replicas of tablet 4aa24b98d69a4b519947558a52e318b5
I20260319 16:36:29.832522 8909 tablet_service.cc:1558] Processing DeleteTablet for tablet 4aa24b98d69a4b519947558a52e318b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:29 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:59958
I20260319 16:36:29.832523 8777 tablet_service.cc:1558] Processing DeleteTablet for tablet 4aa24b98d69a4b519947558a52e318b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:29 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:48000
I20260319 16:36:29.832825 8638 tablet_service.cc:1558] Processing DeleteTablet for tablet 4aa24b98d69a4b519947558a52e318b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:29 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:46436
I20260319 16:36:29.832893 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 8553
I20260319 16:36:29.833482 9003 tablet_replica.cc:333] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: stopping tablet replica
I20260319 16:36:29.833664 9004 tablet_replica.cc:333] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: stopping tablet replica
I20260319 16:36:29.833760 9004 raft_consensus.cc:2243] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:36:29.833869 9004 raft_consensus.cc:2272] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:29.834213 9004 ts_tablet_manager.cc:1916] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:29.835717 9004 ts_tablet_manager.cc:1929] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:29.835784 9004 log.cc:1199] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/4aa24b98d69a4b519947558a52e318b5
I20260319 16:36:29.836027 9004 ts_tablet_manager.cc:1950] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Deleting consensus metadata
I20260319 16:36:29.836362 8500 catalog_manager.cc:5002] TS 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335): tablet 4aa24b98d69a4b519947558a52e318b5 (table test-table [id=2c306ff81c254d0480afd265eef12ba8]) successfully deleted
I20260319 16:36:29.838174 9003 raft_consensus.cc:2243] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:36:29.838336 9003 raft_consensus.cc:2272] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:29.838704 9003 ts_tablet_manager.cc:1916] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260319 16:36:29.838804 8501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:42519 (error 108)
I20260319 16:36:29.839296 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 8709
I20260319 16:36:29.839886 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.839385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42519 (local address 127.0.0.1:46448)
0319 16:36:29.839812 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.839853 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
W20260319 16:36:29.840361 8501 catalog_manager.cc:4729] TS 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4aa24b98d69a4b519947558a52e318b5: Network error: Client connection negotiation failed: client connection to 127.24.187.193:42519: connect: Connection refused (error 111)
I20260319 16:36:29.844398 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 8846
I20260319 16:36:29.844553 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:29.844445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43453 (local address 127.0.0.1:48018)
0319 16:36:29.844496 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:29.844520 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.194:43453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260319 16:36:29.848776 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 8482
I20260319 16:36:29.853596 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39639
--webserver_interface=127.24.187.254
--webserver_port=33401
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:29.926997 9007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:29.927157 9007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:29.927176 9007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:29.928644 9007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:29.928689 9007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:29.928701 9007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:29.928711 9007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:29.928721 9007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:29.930387 9007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=33401
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:29.930569 9007 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:29.930732 9007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:29.932930 9015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:29.932930 9013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.933084 9007 server_base.cc:1060] running on GCE node
W20260319 16:36:29.933017 9012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:29.933393 9007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:29.933609 9007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:29.934733 9007 hybrid_clock.cc:648] HybridClock initialized: now 1773938189934718 us; error 34 us; skew 500 ppm
Mar 19 16:36:29 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938189, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:29.936878 9007 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:29.937398 9007 webserver.cc:492] Webserver started at http://127.24.187.254:33401/ using document root <none> and password file <none>
I20260319 16:36:29.937564 9007 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:29.937598 9007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:29.938735 9007 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:29.939317 9022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:29.939495 9007 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:36:29.939561 9007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "f8e773f5586ac879160192c194c0e75a"
server_key_iv: "496d381593d3556da4b7a70b47f5a891"
server_key_version: "encryptionkey@0"
I20260319 16:36:29.939808 9007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:29.954011 9007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:29.956012 9007 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:29.956122 9007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:29.959841 9007 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39639
I20260319 16:36:29.959874 9074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39639 every 8 connection(s)
I20260319 16:36:29.960166 9007 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:29.961225 9075 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:36:29.961808 9075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Bootstrap starting.
I20260319 16:36:29.967829 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9007
I20260319 16:36:29.968101 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:42519
--local_ip_for_outbound_sockets=127.24.187.193
--tserver_master_addrs=127.24.187.254:39639
--webserver_port=39657
--webserver_interface=127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:29.970335 9075 log.cc:826] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:29.972301 9075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:36:29.972551 9075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6: Bootstrap complete.
I20260319 16:36:29.974553 9075 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.974743 9075 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa3a6209d3f4b1396b7f1c726b2e3b6, State: Initialized, Role: FOLLOWER
I20260319 16:36:29.974871 9075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.974939 9075 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:36:29.974982 9075 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:36:29.975025 9075 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 2 FOLLOWER]: Advancing to term 3
I20260319 16:36:29.975829 9075 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.975939 9075 leader_election.cc:304] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa3a6209d3f4b1396b7f1c726b2e3b6; no voters:
I20260319 16:36:29.976101 9075 leader_election.cc:290] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260319 16:36:29.976150 9079 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260319 16:36:29.976257 9079 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [term 3 LEADER]: Becoming Leader. State: Replica: 5fa3a6209d3f4b1396b7f1c726b2e3b6, State: Running, Role: LEADER
I20260319 16:36:29.976326 9075 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:29.976334 9079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } }
I20260319 16:36:29.976643 9079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa3a6209d3f4b1396b7f1c726b2e3b6. Latest consensus state: current_term: 3 leader_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } } }
I20260319 16:36:29.976714 9079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:29.976833 9081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa3a6209d3f4b1396b7f1c726b2e3b6" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39639 } } }
I20260319 16:36:29.976888 9081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa3a6209d3f4b1396b7f1c726b2e3b6 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:29.977152 9084 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:29.977897 9084 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:29.978010 9084 catalog_manager.cc:1269] Loaded cluster ID: be663a29f6d949d986c9f02850e0c161
I20260319 16:36:29.978041 9084 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
F20260319 16:36:29.978231 9084 catalog_manager.cc:1476] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa0e1720dcd google::LogMessage::Fail() at ??:0
@ 0x7fa0e1724b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa0e17207cc google::LogMessage::Flush() at ??:0
@ 0x7fa0e1721f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa0e3f59b86 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa0e3f642e4 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa0e285d550 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa0e28531b7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa0e2b4a6db start_thread at ??:0
@ 0x7fa0e0c7561f clone at ??:0
W20260319 16:36:30.056135 9077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:30.056322 9077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:30.056346 9077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:30.057880 9077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:30.057924 9077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:30.057950 9077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:36:30.059577 9077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:42519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=39657
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:30.059785 9077 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:30.059949 9077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:30.062449 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:30.062552 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:30.062449 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:30.063009 9077 server_base.cc:1060] running on GCE node
I20260319 16:36:30.063172 9077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:30.063355 9077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:30.064527 9077 hybrid_clock.cc:648] HybridClock initialized: now 1773938190064507 us; error 32 us; skew 500 ppm
Mar 19 16:36:30 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938190, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:30.066960 9077 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:36:30.067485 9077 webserver.cc:492] Webserver started at http://127.24.187.193:39657/ using document root <none> and password file <none>
I20260319 16:36:30.067653 9077 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:30.067689 9077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:30.068835 9077 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:36:30.069406 9121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:30.069576 9077 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:36:30.069655 9077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "3a974ef1544f2ff83744cd577d28dc3f"
server_key_iv: "136cfc55e299b726b3b971b391dc6ad8"
server_key_version: "encryptionkey@0"
I20260319 16:36:30.070012 9077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:30.085098 9077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:30.087282 9077 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:30.087441 9077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:30.087695 9077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:30.088173 9128 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260319 16:36:30.088986 9077 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260319 16:36:30.089047 9077 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:30.089079 9077 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260319 16:36:30.089855 9077 ts_tablet_manager.cc:616] Registered 1 tablets
I20260319 16:36:30.089898 9077 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:30.089955 9128 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap starting.
I20260319 16:36:30.096949 9077 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:42519
I20260319 16:36:30.097131 9235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:42519 every 8 connection(s)
I20260319 16:36:30.097349 9077 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:36:30.099484 9128 log.cc:826] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:30.101136 9128 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:36:30.101388 9128 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Bootstrap complete.
I20260319 16:36:30.102154 9128 ts_tablet_manager.cc:1403] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20260319 16:36:30.103060 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9077
I20260319 16:36:30.102950 9128 raft_consensus.cc:359] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:30.103197 9128 raft_consensus.cc:740] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1cfc6aad0e44b4bd8da5cfa9818d38, State: Initialized, Role: FOLLOWER
I20260319 16:36:30.103322 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:43453
--local_ip_for_outbound_sockets=127.24.187.194
--tserver_master_addrs=127.24.187.254:39639
--webserver_port=39543
--webserver_interface=127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:36:30.103331 9128 consensus_queue.cc:260] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:30.103647 9128 ts_tablet_manager.cc:1434] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20260319 16:36:30.163636 9238 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:36:30.098023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:52527)
0319 16:36:30.098219 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.098256 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:30.098845 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:30.163492 (+ 64647us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
W20260319 16:36:30.163892 9236 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:39639 (0 consecutive failures): Network error: Failed to ping master at 127.24.187.254:39639: Client connection negotiation failed: client connection to 127.24.187.254:39639: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260319 16:36:30.164232 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.164131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:59395)
0319 16:36:30.164178 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.164210 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:36:30.164577 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.164502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:45645)
0319 16:36:30.164540 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.164561 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260319 16:36:30.164686 9236 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260319 16:36:30.178500 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:30.178669 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:30.178691 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:30.180238 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:30.180284 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:30.180308 9241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:36:30.181953 9241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:43453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=39543
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:30.182128 9241 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:30.182262 9241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:30.184501 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:30.184506 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:30.184549 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:30.184863 9241 server_base.cc:1060] running on GCE node
I20260319 16:36:30.185043 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:30.185241 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:30.186386 9241 hybrid_clock.cc:648] HybridClock initialized: now 1773938190186364 us; error 28 us; skew 500 ppm
Mar 19 16:36:30 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938190, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:30.188747 9241 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:36:30.189299 9241 webserver.cc:492] Webserver started at http://127.24.187.194:39543/ using document root <none> and password file <none>
I20260319 16:36:30.189473 9241 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:30.189512 9241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:30.190743 9241 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:30.191459 9256 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:30.191664 9241 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:30.191725 9241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "cbea135349c1477c9fd86ec60a7b9f75"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "2387295af3a0aad324f22b53baf4c7ef"
server_key_iv: "476dcf620ec9b63ae1846ef581c9ade2"
server_key_version: "encryptionkey@0"
I20260319 16:36:30.191991 9241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:30.198891 9241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:30.200656 9241 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:30.200771 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:30.200963 9241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:30.201330 9263 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260319 16:36:30.202152 9241 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260319 16:36:30.202199 9241 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:30.202231 9241 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260319 16:36:30.202745 9241 ts_tablet_manager.cc:616] Registered 1 tablets
I20260319 16:36:30.202776 9241 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:30.202842 9263 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Bootstrap starting.
I20260319 16:36:30.204205 9263 log.cc:826] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:30.206290 9263 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:36:30.206538 9263 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Bootstrap complete.
I20260319 16:36:30.206674 9263 ts_tablet_manager.cc:1403] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260319 16:36:30.207907 9263 raft_consensus.cc:359] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:30.208339 9263 raft_consensus.cc:740] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbea135349c1477c9fd86ec60a7b9f75, State: Initialized, Role: FOLLOWER
I20260319 16:36:30.208488 9263 consensus_queue.cc:260] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:30.208734 9263 ts_tablet_manager.cc:1434] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260319 16:36:30.208892 9241 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:43453
I20260319 16:36:30.208971 9372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:43453 every 8 connection(s)
I20260319 16:36:30.209177 9241 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:36:30.210004 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.209684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:42705)
0319 16:36:30.209858 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.209921 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
W20260319 16:36:30.210191 9373 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:39639 (0 consecutive failures): Network error: Failed to ping master at 127.24.187.254:39639: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
I20260319 16:36:30.210443 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.210362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:51269)
0319 16:36:30.210401 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.210424 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260319 16:36:30.210691 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.210624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:57155)
0319 16:36:30.210656 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.210676 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260319 16:36:30.210770 9373 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260319 16:36:30.218300 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9241
I20260319 16:36:30.218571 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:44335
--local_ip_for_outbound_sockets=127.24.187.195
--tserver_master_addrs=127.24.187.254:39639
--webserver_port=39355
--webserver_interface=127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:30.293205 9376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:30.293391 9376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:30.293414 9376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:30.294939 9376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:30.294979 9376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:30.295006 9376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:36:30.296653 9376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:44335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=39355
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:30.296838 9376 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:30.296978 9376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:30.299252 9384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:30.299247 9382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:30.299247 9381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:30.299518 9376 server_base.cc:1060] running on GCE node
I20260319 16:36:30.299719 9376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:30.299932 9376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:30.301088 9376 hybrid_clock.cc:648] HybridClock initialized: now 1773938190301071 us; error 32 us; skew 500 ppm
Mar 19 16:36:30 dist-test-slave-svn8 krb5kdc[7925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938190, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:30.303561 9376 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:36:30.304080 9376 webserver.cc:492] Webserver started at http://127.24.187.195:39355/ using document root <none> and password file <none>
I20260319 16:36:30.304245 9376 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:30.304281 9376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:30.305531 9376 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:30.306201 9391 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:30.306465 9376 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:36:30.306545 9376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "1d1d6bd90ed04dee8e87343043731e47"
format_stamp: "Formatted at 2026-03-19 16:36:28 on dist-test-slave-svn8"
server_key: "733f84cfc19a2e279ea06df0779025b5"
server_key_iv: "19ce76d7d0fc8dac4aefdf5bd76b210e"
server_key_version: "encryptionkey@0"
I20260319 16:36:30.306813 9376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:30.321309 9376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:30.322942 9376 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:30.323063 9376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:30.323269 9376 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:30.323589 9376 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:30.323621 9376 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:30.323649 9376 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:30.323666 9376 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:30.329159 9376 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:44335
I20260319 16:36:30.329208 9504 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:44335 every 8 connection(s)
I20260319 16:36:30.329492 9376 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:36:30.330399 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.330062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:46219)
0319 16:36:30.330189 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.330258 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
W20260319 16:36:30.330571 9505 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:39639 (0 consecutive failures): Network error: Failed to ping master at 127.24.187.254:39639: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
I20260319 16:36:30.330852 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.330756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:49819)
0319 16:36:30.330805 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.330831 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260319 16:36:30.331084 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:30.331024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:40189)
0319 16:36:30.331053 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:30.331071 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260319 16:36:30.331158 9505 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260319 16:36:30.333010 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9376
I20260319 16:36:31.165252 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.165053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:50703)
0319 16:36:31.165175 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.165206 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260319 16:36:31.211336 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.211104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:34657)
0319 16:36:31.211254 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.211294 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260319 16:36:31.331600 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.331416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:33349)
0319 16:36:31.331518 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.331550 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260319 16:36:31.392683 9512 raft_consensus.cc:493] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:36:31.392794 9512 raft_consensus.cc:515] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:31.393108 9512 leader_election.cc:290] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cbea135349c1477c9fd86ec60a7b9f75 (127.24.187.194:43453), 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335)
Mar 19 16:36:31 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938190, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.195@KRBTEST.COM
Mar 19 16:36:31 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938190, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.194@KRBTEST.COM
I20260319 16:36:31.406251 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.393403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:58361 (local address 127.24.187.195:44335)
0319 16:36:31.393521 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:31.393548 (+ 27us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:31.393561 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:36:31.394110 (+ 549us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:31.394114 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:31.394149 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:31.394233 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:31.394968 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.396404 (+ 1436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:31.397044 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.397282 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:31.402904 (+ 5622us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:31.402934 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:31.402934 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:31.402957 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:31.404837 (+ 1880us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:31.405288 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:31.405291 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:31.405291 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:31.405340 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:31.405504 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:31.405505 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:31.405505 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:31.405698 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:31.405812 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:31.406047 (+ 235us) server_negotiation.cc:300] Negotiation successful
0319 16:36:31.406154 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260319 16:36:31.406244 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.393256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:43453 (local address 127.24.187.193:55039)
0319 16:36:31.393404 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.393415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:31.393493 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:31.393998 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:31.394000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:31.394455 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:31.394844 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:31.394850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.396319 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:31.396320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:31.396865 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:31.396867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.397005 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:31.401095 (+ 4090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:31.401100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:31.403031 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:31.404954 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:31.404955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:31.404955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:31.405204 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:31.405419 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:31.405420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:31.405420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:31.405457 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:31.405781 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:31.405787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:31.405896 (+ 109us) client_negotiation.cc:769] Sending connection context
0319 16:36:31.405987 (+ 91us) client_negotiation.cc:240] Negotiation successful
0319 16:36:31.406137 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260319 16:36:31.406255 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.393365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:55039 (local address 127.24.187.194:43453)
0319 16:36:31.393490 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:31.393499 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:31.393525 (+ 26us) server_negotiation.cc:408] Connection header received
0319 16:36:31.393648 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:31.393650 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:31.393693 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:31.393830 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:31.394964 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.396209 (+ 1245us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:31.396963 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.397227 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:31.403129 (+ 5902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:31.403139 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:31.403139 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:31.403159 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:31.404868 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:31.405279 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:31.405282 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:31.405282 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:31.405340 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:31.405511 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:31.405512 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:31.405512 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:31.405697 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:31.405805 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:31.406043 (+ 238us) server_negotiation.cc:300] Negotiation successful
0319 16:36:31.406142 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
I20260319 16:36:31.406337 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.393288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:44335 (local address 127.24.187.193:58361)
0319 16:36:31.393428 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.393437 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:31.393493 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:31.394241 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:31.394242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:31.394443 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:31.394844 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:31.394850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.396482 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:31.396483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:31.396954 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:31.396956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:31.397010 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:31.401073 (+ 4063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:31.401100 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:31.402752 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:31.404918 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:31.404925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:31.404927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:31.405211 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:31.405397 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:31.405397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:31.405398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:31.405451 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:31.405792 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:31.405794 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:31.405895 (+ 101us) client_negotiation.cc:769] Sending connection context
0319 16:36:31.405986 (+ 91us) client_negotiation.cc:240] Negotiation successful
0319 16:36:31.406227 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":105,"thread_start_us":50,"threads_started":1}
I20260319 16:36:31.406767 9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4aa24b98d69a4b519947558a52e318b5" candidate_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cbea135349c1477c9fd86ec60a7b9f75" is_pre_election: true
I20260319 16:36:31.406929 9326 raft_consensus.cc:2468] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1cfc6aad0e44b4bd8da5cfa9818d38 in term 1.
I20260319 16:36:31.407178 9122 leader_election.cc:304] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1cfc6aad0e44b4bd8da5cfa9818d38, cbea135349c1477c9fd86ec60a7b9f75; no voters:
I20260319 16:36:31.407320 9512 raft_consensus.cc:2804] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260319 16:36:31.407392 9512 raft_consensus.cc:493] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:36:31.407421 9512 raft_consensus.cc:3060] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 1 FOLLOWER]: Advancing to term 2
W20260319 16:36:31.407558 9123 leader_election.cc:343] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335): Not found: Tablet not found: 4aa24b98d69a4b519947558a52e318b5
I20260319 16:36:31.408466 9512 raft_consensus.cc:515] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:31.408588 9512 leader_election.cc:290] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [CANDIDATE]: Term 2 election: Requested vote from peers cbea135349c1477c9fd86ec60a7b9f75 (127.24.187.194:43453), 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335)
I20260319 16:36:31.408762 9326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4aa24b98d69a4b519947558a52e318b5" candidate_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cbea135349c1477c9fd86ec60a7b9f75"
I20260319 16:36:31.408846 9326 raft_consensus.cc:3060] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 1 FOLLOWER]: Advancing to term 2
W20260319 16:36:31.408892 9123 leader_election.cc:343] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335): Not found: Tablet not found: 4aa24b98d69a4b519947558a52e318b5
I20260319 16:36:31.410317 9326 raft_consensus.cc:2468] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1cfc6aad0e44b4bd8da5cfa9818d38 in term 2.
I20260319 16:36:31.410495 9122 leader_election.cc:304] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e1cfc6aad0e44b4bd8da5cfa9818d38, cbea135349c1477c9fd86ec60a7b9f75; no voters: 1d1d6bd90ed04dee8e87343043731e47
I20260319 16:36:31.410612 9512 raft_consensus.cc:2804] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 2 FOLLOWER]: Leader election won for term 2
I20260319 16:36:31.410739 9512 raft_consensus.cc:697] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [term 2 LEADER]: Becoming Leader. State: Replica: 6e1cfc6aad0e44b4bd8da5cfa9818d38, State: Running, Role: LEADER
I20260319 16:36:31.410847 9512 consensus_queue.cc:237] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:31.411104 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.410932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:57209)
0319 16:36:31.411049 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.411081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260319 16:36:31.805637 9123 consensus_peers.cc:597] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 -> Peer 1d1d6bd90ed04dee8e87343043731e47 (127.24.187.195:44335): Couldn't send request to peer 1d1d6bd90ed04dee8e87343043731e47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa24b98d69a4b519947558a52e318b5. This is attempt 1: this message will repeat every 5th retry.
I20260319 16:36:31.994792 9326 raft_consensus.cc:1275] T 4aa24b98d69a4b519947558a52e318b5 P cbea135349c1477c9fd86ec60a7b9f75 [term 2 FOLLOWER]: Refusing update from remote peer 6e1cfc6aad0e44b4bd8da5cfa9818d38: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260319 16:36:31.995129 9512 consensus_queue.cc:1048] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260319 16:36:31.996192 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.996038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:44041)
0319 16:36:31.996126 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.996157 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260319 16:36:31.996953 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.996718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:58899)
0319 16:36:31.996870 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.996908 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":53,"threads_started":1}
I20260319 16:36:31.997051 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:31.996965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:35581)
0319 16:36:31.997006 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:31.997030 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:36:32.229274 9525 ts_tablet_manager.cc:933] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Initiating tablet copy from peer 6e1cfc6aad0e44b4bd8da5cfa9818d38 (127.24.187.193:42519)
I20260319 16:36:32.229511 9525 tablet_copy_client.cc:323] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: tablet copy: Beginning tablet copy session from remote peer at address 127.24.187.193:42519
Mar 19 16:36:32 dist-test-slave-svn8 krb5kdc[7925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938190, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.193@KRBTEST.COM
I20260319 16:36:32.236730 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:32.229885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:59933 (local address 127.24.187.193:42519)
0319 16:36:32.230033 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:32.230039 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:32.230071 (+ 32us) server_negotiation.cc:408] Connection header received
0319 16:36:32.230139 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:32.230141 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:32.230155 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:32.230213 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:32.230833 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:32.231473 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:32.231985 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:32.232132 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:32.233978 (+ 1846us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:32.233986 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:32.233987 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:32.233993 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:32.235530 (+ 1537us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:32.236024 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:32.236028 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:32.236028 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:32.236065 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:32.236198 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:32.236199 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:32.236200 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:32.236314 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:32.236395 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:32.236539 (+ 144us) server_negotiation.cc:300] Negotiation successful
0319 16:36:32.236642 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260319 16:36:32.236748 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:32.229797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42519 (local address 127.24.187.195:59933)
0319 16:36:32.229999 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:32.230015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:32.230088 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:32.230228 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:32.230232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:32.230591 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:32.230770 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:32.230773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:32.231559 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:32.231560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:32.231915 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:32.231917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:32.231995 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:32.232501 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:32.232506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:32.233884 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:32.235629 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:32.235713 (+ 84us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:32.235718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:32.235917 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:32.236121 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:32.236122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:32.236123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:32.236151 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:32.236370 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:32.236375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:32.236432 (+ 57us) client_negotiation.cc:769] Sending connection context
0319 16:36:32.236502 (+ 70us) client_negotiation.cc:240] Negotiation successful
0319 16:36:32.236611 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
I20260319 16:36:32.237114 9210 tablet_copy_service.cc:140] P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Received BeginTabletCopySession request for tablet 4aa24b98d69a4b519947558a52e318b5 from peer 1d1d6bd90ed04dee8e87343043731e47 ({username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:59933)
I20260319 16:36:32.237181 9210 tablet_copy_service.cc:161] P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Beginning new tablet copy session on tablet 4aa24b98d69a4b519947558a52e318b5 from peer 1d1d6bd90ed04dee8e87343043731e47 at {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:59933: session id = 1d1d6bd90ed04dee8e87343043731e47-4aa24b98d69a4b519947558a52e318b5
I20260319 16:36:32.237636 9210 tablet_copy_source_session.cc:215] T 4aa24b98d69a4b519947558a52e318b5 P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Tablet Copy: opened 0 blocks and 1 log segments
I20260319 16:36:32.238483 9525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa24b98d69a4b519947558a52e318b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:32.240661 9525 tablet_copy_client.cc:806] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: tablet copy: Starting download of 0 data blocks...
I20260319 16:36:32.240809 9525 tablet_copy_client.cc:670] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: tablet copy: Starting download of 1 WAL segments...
I20260319 16:36:32.241828 9525 tablet_copy_client.cc:538] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260319 16:36:32.242573 9525 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Bootstrap starting.
I20260319 16:36:32.244124 9525 log.cc:826] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:32.246289 9525 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:36:32.246497 9525 tablet_bootstrap.cc:492] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Bootstrap complete.
I20260319 16:36:32.246609 9525 ts_tablet_manager.cc:1403] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260319 16:36:32.247767 9525 raft_consensus.cc:359] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:32.247900 9525 raft_consensus.cc:740] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1d6bd90ed04dee8e87343043731e47, State: Initialized, Role: FOLLOWER
I20260319 16:36:32.247990 9525 consensus_queue.cc:260] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbea135349c1477c9fd86ec60a7b9f75" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 43453 } } peers { permanent_uuid: "1d1d6bd90ed04dee8e87343043731e47" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 44335 } } peers { permanent_uuid: "6e1cfc6aad0e44b4bd8da5cfa9818d38" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42519 } }
I20260319 16:36:32.248217 9525 ts_tablet_manager.cc:1434] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:36:32.248507 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:32.248412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:35867)
0319 16:36:32.248454 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:32.248481 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:36:32.248654 9210 tablet_copy_service.cc:342] P 6e1cfc6aad0e44b4bd8da5cfa9818d38: Request end of tablet copy session 1d1d6bd90ed04dee8e87343043731e47-4aa24b98d69a4b519947558a52e318b5 received from {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:59933
I20260319 16:36:32.248728 9210 tablet_copy_service.cc:434] P 6e1cfc6aad0e44b4bd8da5cfa9818d38: ending tablet copy session 1d1d6bd90ed04dee8e87343043731e47-4aa24b98d69a4b519947558a52e318b5 on tablet 4aa24b98d69a4b519947558a52e318b5 with peer 1d1d6bd90ed04dee8e87343043731e47
I20260319 16:36:32.248878 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:32.248729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:42513)
0319 16:36:32.248776 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:32.248839 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:36:32.636334 9459 raft_consensus.cc:1217] T 4aa24b98d69a4b519947558a52e318b5 P 1d1d6bd90ed04dee8e87343043731e47 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260319 16:36:32.636474 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:32.636176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:44721)
0319 16:36:32.636383 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:32.636425 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":63,"threads_started":1}
I20260319 16:36:32.636893 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:32.636758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:49867)
0319 16:36:32.636829 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:32.636864 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260319 16:36:32.997697 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:32.997470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:37273)
0319 16:36:32.997596 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:32.997648 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260319 16:36:33.637185 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:33.636933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:32903)
0319 16:36:33.637059 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:33.637126 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260319 16:36:33.637508 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:33.637303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:39655)
0319 16:36:33.637419 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:33.637450 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260319 16:36:33.998360 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:33.998148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:41889)
0319 16:36:33.998268 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:33.998309 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260319 16:36:34.637890 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:34.637620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:55415)
0319 16:36:34.637792 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:34.637829 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20260319 16:36:34.638165 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:34.637970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:39357)
0319 16:36:34.638079 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:34.638117 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260319 16:36:34.999043 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:34.998841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:60181)
0319 16:36:34.998956 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:34.998995 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260319 16:36:35.638821 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:35.638562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:57607)
0319 16:36:35.638723 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:35.638764 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260319 16:36:35.638841 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:35.638646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:41139)
0319 16:36:35.638767 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:35.638799 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260319 16:36:35.639171 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:35.639090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:38293)
0319 16:36:35.639130 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:35.639154 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:36:35.999790 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:35.999580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:53493)
0319 16:36:35.999701 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:35.999741 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260319 16:36:36.639616 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:36.639323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:43757)
0319 16:36:36.639515 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:36.639551 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":126,"threads_started":1}
I20260319 16:36:36.639835 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:36.639623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:47509)
0319 16:36:36.639757 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:36.639792 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260319 16:36:37.000532 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:37.000263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:45557)
0319 16:36:37.000429 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:37.000472 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260319 16:36:37.640300 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:37.640089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:58913)
0319 16:36:37.640213 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:37.640246 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260319 16:36:37.640473 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:37.640255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:53125)
0319 16:36:37.640379 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:37.640415 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260319 16:36:38.001322 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:38.001059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:35751)
0319 16:36:38.001216 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:38.001259 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260319 16:36:38.641016 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:38.640797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:54177)
0319 16:36:38.640934 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:38.640966 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260319 16:36:38.641093 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:38.640882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:54591)
0319 16:36:38.641017 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:38.641049 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260319 16:36:39.002103 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:39.001869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:56445)
0319 16:36:39.002013 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:39.002055 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260319 16:36:39.641873 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:39.641601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:42373)
0319 16:36:39.641779 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:39.641828 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260319 16:36:39.641870 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:39.641604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:33657)
0319 16:36:39.641778 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:39.641813 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260319 16:36:40.002818 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:40.002599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:35601)
0319 16:36:40.002737 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:40.002769 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260319 16:36:40.642565 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:40.642330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:46809)
0319 16:36:40.642473 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:40.642509 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260319 16:36:40.642565 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:40.642328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:54917)
0319 16:36:40.642472 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:40.642509 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260319 16:36:41.003618 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:41.003327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:54307)
0319 16:36:41.003523 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:41.003568 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260319 16:36:41.643155 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:41.642976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:51941)
0319 16:36:41.643079 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:41.643108 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260319 16:36:41.643218 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:41.643046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:44059)
0319 16:36:41.643149 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:41.643175 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
I20260319 16:36:42.004276 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:42.004058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:57137)
0319 16:36:42.004190 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:42.004227 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260319 16:36:42.643828 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:42.643610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:46391)
0319 16:36:42.643743 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:42.643780 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260319 16:36:42.643841 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:42.643654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:41629)
0319 16:36:42.643770 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:42.643801 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260319 16:36:43.004997 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:43.004786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:45267)
0319 16:36:43.004908 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:43.004946 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260319 16:36:43.644542 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:43.644284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:46447)
0319 16:36:43.644447 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:43.644485 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260319 16:36:43.644614 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:43.644406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:51171)
0319 16:36:43.644536 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:43.644570 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260319 16:36:44.005784 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:44.005526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:57897)
0319 16:36:44.005651 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:44.005691 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260319 16:36:44.645408 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:44.645015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.193:37557)
0319 16:36:44.645318 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:44.645350 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":57,"threads_started":1}
I20260319 16:36:44.645429 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:44.645199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.195:47109)
0319 16:36:44.645327 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:44.645369 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
I20260319 16:36:45.006490 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:45.006264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39639 (local address 127.24.187.194:43089)
0319 16:36:45.006389 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:45.006442 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260319 16:36:45.343453 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 9077
I20260319 16:36:45.348440 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 9241
I20260319 16:36:45.353099 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 9376
2026-03-19T16:36:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19087 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:36:45 dist-test-slave-svn8 krb5kdc[9572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:36:45 dist-test-slave-svn8 krb5kdc[9572](info): set up 2 sockets
Mar 19 16:36:45 dist-test-slave-svn8 krb5kdc[9572](info): commencing operation
krb5kdc: starting...
W20260319 16:36:47.186355 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:36:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:47.323985 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35859
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:47.398962 9588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:47.399161 9588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:47.399184 9588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:47.400767 9588 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260319 16:36:47.400810 9588 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260319 16:36:47.400821 9588 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260319 16:36:47.400831 9588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:36:47.400843 9588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:47.400854 9588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:47.400867 9588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:36:47.400887 9588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:36:47.402592 9588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:47.402809 9588 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:47.402992 9588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:36:47.405467 9588 server_base.cc:1060] running on GCE node
W20260319 16:36:47.405556 9594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:47.405664 9596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:47.405568 9593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:47.405946 9588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:47.406138 9588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:47.407265 9588 hybrid_clock.cc:648] HybridClock initialized: now 1773938207407250 us; error 28 us; skew 500 ppm
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:47.409659 9588 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:36:47.410207 9588 webserver.cc:492] Webserver started at http://127.24.187.254:33593/ using document root <none> and password file <none>
I20260319 16:36:47.410370 9588 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:47.410403 9588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:47.410503 9588 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:47.411479 9588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "2c657bb394ec4c4e80183c288ca36675"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "0e2561e04b199aafe1798543569ff613"
server_key_iv: "0120a97ab9409584b1b79d9b4de5154d"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.411783 9588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "2c657bb394ec4c4e80183c288ca36675"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "0e2561e04b199aafe1798543569ff613"
server_key_iv: "0120a97ab9409584b1b79d9b4de5154d"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.412844 9588 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:36:47.413426 9603 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.413601 9588 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:36:47.413658 9588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2c657bb394ec4c4e80183c288ca36675"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "0e2561e04b199aafe1798543569ff613"
server_key_iv: "0120a97ab9409584b1b79d9b4de5154d"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.413758 9588 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:47.430382 9588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:47.432058 9588 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:47.432184 9588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:47.435987 9588 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:35859
I20260319 16:36:47.436030 9655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:35859 every 8 connection(s)
I20260319 16:36:47.436303 9588 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:36:47.436827 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:47.438354 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9588
I20260319 16:36:47.438462 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:36:47.438620 25327 external_mini_cluster.cc:1468] Setting key 240f4bca6133b085cb53af697cb5dc39
I20260319 16:36:47.439491 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Bootstrap starting.
I20260319 16:36:47.440132 9656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:47.440428 9656 log.cc:826] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:47.441128 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: No bootstrap required, opened a new log
I20260319 16:36:47.442533 9656 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:36:47.442658 9656 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:47.442685 9656 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c657bb394ec4c4e80183c288ca36675, State: Initialized, Role: FOLLOWER
I20260319 16:36:47.442783 9656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:36:47.442843 9656 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:36:47.442873 9656 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:36:47.442915 9656 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:47.443631 9656 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:36:47.443743 9656 leader_election.cc:304] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c657bb394ec4c4e80183c288ca36675; no voters:
I20260319 16:36:47.443892 9656 leader_election.cc:290] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:36:47.443944 9661 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:47.444036 9661 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 LEADER]: Becoming Leader. State: Replica: 2c657bb394ec4c4e80183c288ca36675, State: Running, Role: LEADER
I20260319 16:36:47.444120 9656 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:36:47.444106 9661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:36:47.444402 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c657bb394ec4c4e80183c288ca36675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } } }
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:47.444552 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:47.444669 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c657bb394ec4c4e80183c288ca36675. Latest consensus state: current_term: 1 leader_uuid: "2c657bb394ec4c4e80183c288ca36675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } } }
I20260319 16:36:47.444722 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: This master's current role is: LEADER
I20260319 16:36:47.445504 9666 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:36:47.446092 9666 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:36:47.447330 9666 catalog_manager.cc:1357] Generated new cluster ID: 3979d3504cba4fd793224eb13bbd3c81
I20260319 16:36:47.447384 9666 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:36:47.448555 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.439087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.24.187.254:35859)
0319 16:36:47.439225 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.439234 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.439251 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:36:47.439547 (+ 296us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.439553 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.439617 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.439717 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:47.440691 (+ 974us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.441568 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.442390 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.442610 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.445162 (+ 2552us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:47.445170 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:47.445171 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:47.445192 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:47.447212 (+ 2020us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.447630 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.447632 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.447633 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.447687 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.447827 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.447828 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.447829 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.448041 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:47.448183 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.448280 (+ 97us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.448404 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260319 16:36:47.450871 9666 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:36:47.451320 9666 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:36:47.460623 9666 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Generated new TSK 0
I20260319 16:36:47.460774 9666 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:47.516273 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:35859
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:47.593592 9684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:47.593791 9684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:47.593816 9684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:47.595289 9684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:47.595342 9684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:47.595367 9684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:36:47.597008 9684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:47.597219 9684 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:47.597398 9684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:47.599728 9692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:47.599848 9689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:47.599738 9690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:47.599773 9684 server_base.cc:1060] running on GCE node
I20260319 16:36:47.600095 9684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:47.600304 9684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:47.601459 9684 hybrid_clock.cc:648] HybridClock initialized: now 1773938207601444 us; error 26 us; skew 500 ppm
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:47.604197 9684 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:36:47.604841 9684 webserver.cc:492] Webserver started at http://127.24.187.193:46055/ using document root <none> and password file <none>
I20260319 16:36:47.605065 9684 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:47.605113 9684 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:47.605221 9684 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:47.606416 9684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "edb0dfbc6ff54005ae3f1c0625649274"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "297f8cbcdd96b0ae9e6db8779b766c4e"
server_key_iv: "e7cb50bcd0aa7c04c6ad929ae51e8dcb"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.606741 9684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "edb0dfbc6ff54005ae3f1c0625649274"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "297f8cbcdd96b0ae9e6db8779b766c4e"
server_key_iv: "e7cb50bcd0aa7c04c6ad929ae51e8dcb"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.608029 9684 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260319 16:36:47.608758 9699 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.608947 9684 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:36:47.609015 9684 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "edb0dfbc6ff54005ae3f1c0625649274"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "297f8cbcdd96b0ae9e6db8779b766c4e"
server_key_iv: "e7cb50bcd0aa7c04c6ad929ae51e8dcb"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.609077 9684 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:47.619225 9684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:47.621006 9684 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:47.621124 9684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:47.621330 9684 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:47.621644 9684 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:47.621685 9684 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.621740 9684 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:47.621763 9684 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.627379 9684 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:44517
I20260319 16:36:47.627440 9812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:44517 every 8 connection(s)
I20260319 16:36:47.627702 9684 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:36:47.631120 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9684
I20260319 16:36:47.631240 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:36:47.631480 25327 external_mini_cluster.cc:1468] Setting key 0355a696f7bc9a84b447925db15c4664
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:47.636653 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.628419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:33493 (local address 127.24.187.254:35859)
0319 16:36:47.628468 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.628470 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.628584 (+ 114us) server_negotiation.cc:408] Connection header received
0319 16:36:47.629071 (+ 487us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.629072 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.629082 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.629134 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:47.630018 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.630527 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.631143 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.631309 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.633979 (+ 2670us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:47.633984 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:47.633984 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:47.633993 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:47.635273 (+ 1280us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.635703 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.635704 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.635705 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.635726 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.636033 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.636034 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.636035 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.636131 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:47.636192 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.636505 (+ 313us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.636586 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:36:47.636845 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.628321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:33493)
0319 16:36:47.628462 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:47.628488 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:47.628984 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:47.629188 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:47.629191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:47.629483 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:47.629861 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.629867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.630635 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.630636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:47.631047 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.631050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.631174 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.631649 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:47.631665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:47.633788 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:47.635353 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:47.635359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:47.635361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:47.635631 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:47.635781 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:47.635781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:47.635782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:47.635863 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:47.636199 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:47.636205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:47.636368 (+ 163us) client_negotiation.cc:769] Sending connection context
0319 16:36:47.636497 (+ 129us) client_negotiation.cc:240] Negotiation successful
0319 16:36:47.636639 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260319 16:36:47.637383 9813 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35859
I20260319 16:36:47.637471 9813 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:47.637657 9813 heartbeater.cc:507] Master 127.24.187.254:35859 requested a full tablet report, sending...
I20260319 16:36:47.638156 9620 ts_manager.cc:194] Registered new tserver with Master: edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517)
I20260319 16:36:47.638788 9620 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:33493
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:47.680299 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:35859
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:47.755420 9820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:47.755585 9820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:47.755615 9820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:47.757037 9820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:47.757078 9820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:47.757110 9820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:36:47.758700 9820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:47.758894 9820 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:47.759049 9820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:47.761675 9826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:47.761799 9820 server_base.cc:1060] running on GCE node
W20260319 16:36:47.761831 9828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:47.761672 9825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:47.762121 9820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:47.762369 9820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:47.763522 9820 hybrid_clock.cc:648] HybridClock initialized: now 1773938207763504 us; error 29 us; skew 500 ppm
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:47.765784 9820 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:36:47.766228 9820 webserver.cc:492] Webserver started at http://127.24.187.194:34891/ using document root <none> and password file <none>
I20260319 16:36:47.766376 9820 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:47.766404 9820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:47.766480 9820 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:47.767343 9820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "45b0139f4a5e4649b1988c454b5f4fe7"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "fe4268c1f7cac0d4aab3d9b889c786ce"
server_key_iv: "34bc380cf34f475832ea404d79915daf"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.767596 9820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "45b0139f4a5e4649b1988c454b5f4fe7"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "fe4268c1f7cac0d4aab3d9b889c786ce"
server_key_iv: "34bc380cf34f475832ea404d79915daf"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.768661 9820 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:36:47.769368 9835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.769580 9820 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:47.769641 9820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "45b0139f4a5e4649b1988c454b5f4fe7"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "fe4268c1f7cac0d4aab3d9b889c786ce"
server_key_iv: "34bc380cf34f475832ea404d79915daf"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.769694 9820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:47.782902 9820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:47.784958 9820 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:47.785070 9820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:47.785252 9820 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:47.785562 9820 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:47.785595 9820 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.785617 9820 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:47.785640 9820 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.791113 9820 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:33141
I20260319 16:36:47.791180 9949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:33141 every 8 connection(s)
I20260319 16:36:47.791433 9820 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:36:47.794931 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9820
I20260319 16:36:47.795049 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:36:47.795434 25327 external_mini_cluster.cc:1468] Setting key d46842ebdde0eafe8099f392a3edace4
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:47.800082 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.792101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:38875 (local address 127.24.187.254:35859)
0319 16:36:47.792175 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.792177 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.792264 (+ 87us) server_negotiation.cc:408] Connection header received
0319 16:36:47.792775 (+ 511us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.792777 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.792788 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.792841 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:47.793653 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.794173 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.794735 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.794872 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.797520 (+ 2648us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:47.797523 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:47.797523 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:47.797530 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:47.798875 (+ 1345us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.799309 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.799310 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.799311 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.799332 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.799519 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.799520 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.799520 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.799618 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:47.799663 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.799947 (+ 284us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.800016 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260319 16:36:47.800274 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.792051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:38875)
0319 16:36:47.792169 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:47.792196 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:47.792677 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:47.792884 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:47.792887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:47.793224 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:47.793547 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.793552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.794239 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.794240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:47.794640 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.794642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.794739 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.795313 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:47.795341 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:47.797429 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:47.798955 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:47.798961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:47.798964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:47.799233 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:47.799392 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:47.799393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:47.799393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:47.799468 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:47.799669 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:47.799674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:47.799820 (+ 146us) client_negotiation.cc:769] Sending connection context
0319 16:36:47.799935 (+ 115us) client_negotiation.cc:240] Negotiation successful
0319 16:36:47.800075 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260319 16:36:47.800799 9950 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35859
I20260319 16:36:47.800892 9950 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:47.801079 9950 heartbeater.cc:507] Master 127.24.187.254:35859 requested a full tablet report, sending...
I20260319 16:36:47.801502 9620 ts_manager.cc:194] Registered new tserver with Master: 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141)
I20260319 16:36:47.802053 9620 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:38875
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:36:47.844648 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:35859
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:36:47.918170 9957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:36:47.918339 9957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:36:47.918370 9957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:36:47.919819 9957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:36:47.919863 9957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:36:47.919896 9957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:36:47.921506 9957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:36:47.921757 9957 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:36:47.921916 9957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:36:47.924209 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:47.924217 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:36:47.924224 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:36:47.924386 9957 server_base.cc:1060] running on GCE node
I20260319 16:36:47.924577 9957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:36:47.924762 9957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:36:47.925880 9957 hybrid_clock.cc:648] HybridClock initialized: now 1773938207925866 us; error 25 us; skew 500 ppm
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:36:47.928038 9957 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:36:47.928489 9957 webserver.cc:492] Webserver started at http://127.24.187.195:35507/ using document root <none> and password file <none>
I20260319 16:36:47.928654 9957 fs_manager.cc:362] Metadata directory not provided
I20260319 16:36:47.928690 9957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:36:47.928787 9957 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:36:47.929664 9957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "28fac62a7cd24d058fb1364cc38476cb"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "819f722c6c60daa2bfc81d57883ecffe"
server_key_iv: "0c31155cf243b49868c9ae733fa686c2"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.929977 9957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "28fac62a7cd24d058fb1364cc38476cb"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "819f722c6c60daa2bfc81d57883ecffe"
server_key_iv: "0c31155cf243b49868c9ae733fa686c2"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.931025 9957 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:36:47.931612 9972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.931766 9957 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:36:47.931821 9957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "28fac62a7cd24d058fb1364cc38476cb"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "819f722c6c60daa2bfc81d57883ecffe"
server_key_iv: "0c31155cf243b49868c9ae733fa686c2"
server_key_version: "encryptionkey@0"
I20260319 16:36:47.931880 9957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:36:47.941977 9957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:36:47.943464 9957 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:36:47.943599 9957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:36:47.943787 9957 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:36:47.944072 9957 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:36:47.944104 9957 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.944132 9957 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:36:47.944156 9957 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:36:47.949759 9957 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:41983
I20260319 16:36:47.949788 10085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:41983 every 8 connection(s)
I20260319 16:36:47.950055 9957 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
Mar 19 16:36:47 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:36:47.958390 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.950925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:50523 (local address 127.24.187.254:35859)
0319 16:36:47.950989 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.950991 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.951353 (+ 362us) server_negotiation.cc:408] Connection header received
0319 16:36:47.951857 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.951859 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.951869 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.951910 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:47.952733 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.953223 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.953799 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.953915 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.956163 (+ 2248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:47.956167 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:47.956167 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:47.956180 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:47.957245 (+ 1065us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.957597 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.957598 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.957598 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.957635 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.957829 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.957829 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.957830 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.957931 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:47.957979 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.958243 (+ 264us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.958310 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260319 16:36:47.958506 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.950670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:50523)
0319 16:36:47.951254 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:47.951282 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:47.951772 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:47.952009 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:47.952013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:47.952334 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:36:47.952632 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.952636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.953311 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.953312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:47.953698 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.953699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.953835 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.954260 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:36:47.954274 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:36:47.956080 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:36:47.957320 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:47.957325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:47.957326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:47.957531 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:47.957683 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:36:47.957684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:36:47.957684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:36:47.957777 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:36:47.958004 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:36:47.958010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:36:47.958119 (+ 109us) client_negotiation.cc:769] Sending connection context
0319 16:36:47.958231 (+ 112us) client_negotiation.cc:240] Negotiation successful
0319 16:36:47.958343 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":67,"threads_started":1}
I20260319 16:36:47.958957 10086 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35859
I20260319 16:36:47.959040 10086 heartbeater.cc:461] Registering TS with master...
I20260319 16:36:47.959134 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 9957
I20260319 16:36:47.959221 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:36:47.959226 10086 heartbeater.cc:507] Master 127.24.187.254:35859 requested a full tablet report, sending...
I20260319 16:36:47.959406 25327 external_mini_cluster.cc:1468] Setting key abb55806464af08895e2377da214e5d4
I20260319 16:36:47.959658 9620 ts_manager.cc:194] Registered new tserver with Master: 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983)
I20260319 16:36:47.960021 9620 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:50523
I20260319 16:36:47.960247 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:36:47.966418 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.961571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.24.187.254:35859)
0319 16:36:47.961617 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.961619 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.961753 (+ 134us) server_negotiation.cc:408] Connection header received
0319 16:36:47.961842 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.961842 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.961850 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.961894 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:36:47.962502 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.962944 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.963533 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.963658 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.964444 (+ 786us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:36:47.964446 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:36:47.964446 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:36:47.964451 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:36:47.965557 (+ 1106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.965851 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.965851 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.965852 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.965870 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:36:47.966036 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:36:47.966037 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:36:47.966037 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:36:47.966136 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:36:47.966210 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.966275 (+ 65us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.966331 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:36:47.967790 9620 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:36:47.968984 9620 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:36:47.974092 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.970851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:33141 (local address 127.0.0.1:46054)
0319 16:36:47.971419 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:47.971429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:47.971490 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:47.971701 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:47.971701 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:47.971703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:47.971887 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.971888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.973185 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.973186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:47.973809 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.973811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.973871 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.973913 (+ 42us) client_negotiation.cc:769] Sending connection context
0319 16:36:47.973956 (+ 43us) client_negotiation.cc:240] Negotiation successful
0319 16:36:47.973996 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":60,"threads_started":1}
I20260319 16:36:47.974198 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.970849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:44517 (local address 127.0.0.1:58900)
0319 16:36:47.970999 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:47.971026 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:47.971087 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:47.971328 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:47.971331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:47.971334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:47.971650 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.971652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.972926 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.972927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:47.973909 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.973912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.973970 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.973974 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:36:47.974001 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:36:47.974037 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260319 16:36:47.974598 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.971004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.24.187.194:33141)
0319 16:36:47.971165 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.971171 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.971542 (+ 371us) server_negotiation.cc:408] Connection header received
0319 16:36:47.971589 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.971591 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.971604 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.971702 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:47.971963 (+ 261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.973077 (+ 1114us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.973895 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.974452 (+ 557us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.974489 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.974512 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.974541 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260319 16:36:47.974833 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.970856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:41983 (local address 127.0.0.1:36326)
0319 16:36:47.971557 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:47.971564 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:47.971631 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:47.972018 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:47.972019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:47.972021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:47.972173 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.972174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.973187 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.973188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:47.974648 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:47.974651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.974712 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.974714 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:36:47.974741 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:36:47.974772 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":84,"threads_started":1}
I20260319 16:36:47.974843 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.971003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.24.187.193:44517)
0319 16:36:47.971164 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.971172 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.971182 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:36:47.971222 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.971224 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.971239 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.971304 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:47.971734 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.972845 (+ 1111us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.974149 (+ 1304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.974655 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.974699 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.974726 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.974769 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260319 16:36:47.975085 9884 tablet_service.cc:1511] Processing CreateTablet for tablet 92ceada3bdd8459dafccbc3eda0ed7e2 (DEFAULT_TABLE table=test-table [id=29a7dcd4767c409c84d92e83cec73473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:47.975353 9884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ceada3bdd8459dafccbc3eda0ed7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:47.975559 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:47.971156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.24.187.195:41983)
0319 16:36:47.971302 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:47.971307 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:47.971861 (+ 554us) server_negotiation.cc:408] Connection header received
0319 16:36:47.971899 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:47.971902 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:47.971916 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:47.972010 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:47.972268 (+ 258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.973036 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:47.974998 (+ 1962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:47.975405 (+ 407us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:47.975444 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:47.975471 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:36:47.975504 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260319 16:36:47.976012 10020 tablet_service.cc:1511] Processing CreateTablet for tablet 92ceada3bdd8459dafccbc3eda0ed7e2 (DEFAULT_TABLE table=test-table [id=29a7dcd4767c409c84d92e83cec73473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:47.976264 10020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ceada3bdd8459dafccbc3eda0ed7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:47.977566 10103 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Bootstrap starting.
I20260319 16:36:47.977900 9747 tablet_service.cc:1511] Processing CreateTablet for tablet 92ceada3bdd8459dafccbc3eda0ed7e2 (DEFAULT_TABLE table=test-table [id=29a7dcd4767c409c84d92e83cec73473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:36:47.978132 9747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ceada3bdd8459dafccbc3eda0ed7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:36:47.978206 10103 tablet_bootstrap.cc:654] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:47.978497 10103 log.cc:826] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:47.979470 10103 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: No bootstrap required, opened a new log
I20260319 16:36:47.979552 10103 ts_tablet_manager.cc:1403] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:47.980790 10106 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap starting.
I20260319 16:36:47.980872 10104 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Bootstrap starting.
I20260319 16:36:47.981055 10103 raft_consensus.cc:359] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:47.981205 10103 raft_consensus.cc:385] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:47.981235 10103 raft_consensus.cc:740] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b0139f4a5e4649b1988c454b5f4fe7, State: Initialized, Role: FOLLOWER
I20260319 16:36:47.981354 10103 consensus_queue.cc:260] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:47.981462 10106 tablet_bootstrap.cc:654] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:47.981530 10104 tablet_bootstrap.cc:654] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Neither blocks nor log segments found. Creating new log.
I20260319 16:36:47.981606 10103 ts_tablet_manager.cc:1434] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:36:47.981755 9950 heartbeater.cc:499] Master 127.24.187.254:35859 was elected leader, sending a full tablet report...
I20260319 16:36:47.981823 10106 log.cc:826] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:47.982095 10104 log.cc:826] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Log is configured to *not* fsync() on all Append() calls
I20260319 16:36:47.982599 10106 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: No bootstrap required, opened a new log
I20260319 16:36:47.982671 10106 ts_tablet_manager.cc:1403] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:36:47.983880 10106 raft_consensus.cc:359] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:47.983994 10106 raft_consensus.cc:385] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:47.984011 10106 raft_consensus.cc:740] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb0dfbc6ff54005ae3f1c0625649274, State: Initialized, Role: FOLLOWER
I20260319 16:36:47.984086 10106 consensus_queue.cc:260] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:47.984299 10106 ts_tablet_manager.cc:1434] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:36:47.984367 9813 heartbeater.cc:499] Master 127.24.187.254:35859 was elected leader, sending a full tablet report...
I20260319 16:36:47.984880 10104 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: No bootstrap required, opened a new log
I20260319 16:36:47.984939 10104 ts_tablet_manager.cc:1403] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260319 16:36:47.986232 10104 raft_consensus.cc:359] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:47.986336 10104 raft_consensus.cc:385] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:36:47.986373 10104 raft_consensus.cc:740] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fac62a7cd24d058fb1364cc38476cb, State: Initialized, Role: FOLLOWER
I20260319 16:36:47.986461 10104 consensus_queue.cc:260] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:47.986675 10104 ts_tablet_manager.cc:1434] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:36:47.986727 10086 heartbeater.cc:499] Master 127.24.187.254:35859 was elected leader, sending a full tablet report...
W20260319 16:36:48.041955 9951 tablet.cc:2379] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:36:48.058097 10111 raft_consensus.cc:493] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:36:48.058238 10111 raft_consensus.cc:515] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:48.058641 10111 leader_election.cc:290] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517), 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141)
I20260319 16:36:48.061044 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:48.058843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:44517 (local address 127.24.187.195:57529)
0319 16:36:48.058910 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:48.058921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:48.058975 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:48.059188 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:48.059189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:48.059192 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:48.059420 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:48.059422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.060173 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:48.060174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:48.060883 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:48.060885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.060940 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:48.060944 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:36:48.060964 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:36:48.060989 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260319 16:36:48.061510 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:48.058882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:33141 (local address 127.24.187.195:52807)
0319 16:36:48.059082 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:48.059092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:36:48.059157 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:36:48.059289 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:36:48.059290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:36:48.059291 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:36:48.059589 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:48.059591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.060498 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:36:48.060499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:36:48.061191 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:36:48.061193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.061268 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:48.061271 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:36:48.061300 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:36:48.061336 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":124,"threads_started":1}
I20260319 16:36:48.061527 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:48.058936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:57529 (local address 127.24.187.193:44517)
0319 16:36:48.058991 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:48.058993 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:48.059024 (+ 31us) server_negotiation.cc:408] Connection header received
0319 16:36:48.059076 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:48.059078 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:48.059087 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:48.059177 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:48.059505 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.060064 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:48.060966 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.061409 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:48.061418 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:48.061442 (+ 24us) server_negotiation.cc:300] Negotiation successful
0319 16:36:48.061472 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260319 16:36:48.061815 9767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ceada3bdd8459dafccbc3eda0ed7e2" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb0dfbc6ff54005ae3f1c0625649274" is_pre_election: true
I20260319 16:36:48.061872 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:48.058976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:52807 (local address 127.24.187.194:33141)
0319 16:36:48.059027 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:48.059029 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:48.059162 (+ 133us) server_negotiation.cc:408] Connection header received
0319 16:36:48.059210 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:48.059211 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:48.059221 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:48.059281 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:36:48.059707 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.060412 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:48.061281 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.061755 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:48.061763 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:48.061786 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:36:48.061817 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:36:48.061956 9767 raft_consensus.cc:2468] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28fac62a7cd24d058fb1364cc38476cb in term 0.
I20260319 16:36:48.062135 9904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ceada3bdd8459dafccbc3eda0ed7e2" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" is_pre_election: true
I20260319 16:36:48.062232 9974 leader_election.cc:304] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fac62a7cd24d058fb1364cc38476cb, edb0dfbc6ff54005ae3f1c0625649274; no voters:
I20260319 16:36:48.062263 9904 raft_consensus.cc:2468] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28fac62a7cd24d058fb1364cc38476cb in term 0.
I20260319 16:36:48.062378 10111 raft_consensus.cc:2804] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:36:48.062449 10111 raft_consensus.cc:493] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:36:48.062482 10111 raft_consensus.cc:3060] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:48.063174 10111 raft_consensus.cc:515] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:48.063294 10111 leader_election.cc:290] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 election: Requested vote from peers edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517), 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141)
I20260319 16:36:48.063506 9904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ceada3bdd8459dafccbc3eda0ed7e2" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b0139f4a5e4649b1988c454b5f4fe7"
I20260319 16:36:48.063520 9767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ceada3bdd8459dafccbc3eda0ed7e2" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb0dfbc6ff54005ae3f1c0625649274"
I20260319 16:36:48.063587 9904 raft_consensus.cc:3060] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:48.063587 9767 raft_consensus.cc:3060] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:36:48.064436 9904 raft_consensus.cc:2468] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28fac62a7cd24d058fb1364cc38476cb in term 1.
I20260319 16:36:48.064436 9767 raft_consensus.cc:2468] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28fac62a7cd24d058fb1364cc38476cb in term 1.
I20260319 16:36:48.064646 9973 leader_election.cc:304] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fac62a7cd24d058fb1364cc38476cb, 45b0139f4a5e4649b1988c454b5f4fe7; no voters:
I20260319 16:36:48.064744 10111 raft_consensus.cc:2804] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:36:48.064911 10111 raft_consensus.cc:697] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 1 LEADER]: Becoming Leader. State: Replica: 28fac62a7cd24d058fb1364cc38476cb, State: Running, Role: LEADER
I20260319 16:36:48.065007 10111 consensus_queue.cc:237] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:36:48.065821 9620 catalog_manager.cc:5671] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195). New cstate: current_term: 1 leader_uuid: "28fac62a7cd24d058fb1364cc38476cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } health_report { overall_health: HEALTHY } } }
I20260319 16:36:48.089818 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:48.087808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.24.187.195:41983)
0319 16:36:48.087867 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:36:48.087869 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:36:48.087876 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:36:48.087920 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:36:48.087921 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:36:48.087936 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:36:48.087982 (+ 46us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:36:48.088216 (+ 234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.088693 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:36:48.089260 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:36:48.089445 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:36:48.089483 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:36:48.089566 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:36:48.089632 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:36:48.089693 (+ 61us) server_negotiation.cc:300] Negotiation successful
0319 16:36:48.089743 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:36:48.091342 9767 raft_consensus.cc:1275] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Refusing update from remote peer 28fac62a7cd24d058fb1364cc38476cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:48.091341 9904 raft_consensus.cc:1275] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Refusing update from remote peer 28fac62a7cd24d058fb1364cc38476cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:36:48.091631 10113 consensus_queue.cc:1048] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:48.091746 10111 consensus_queue.cc:1048] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:36:48.094601 10117 mvcc.cc:204] Tried to move back new op lower bound from 7266050900339761152 to 7266050900234919936. Current Snapshot: MvccSnapshot[applied={T|T < 7266050900339761152 or (T in {7266050900339761152})}]
I20260319 16:36:48.096110 9619 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:48.096190 9619 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:36:48.096854 9619 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Sending DeleteTablet for 3 replicas of tablet 92ceada3bdd8459dafccbc3eda0ed7e2
I20260319 16:36:48.097137 9747 tablet_service.cc:1558] Processing DeleteTablet for tablet 92ceada3bdd8459dafccbc3eda0ed7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:48 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:58900
I20260319 16:36:48.097139 9884 tablet_service.cc:1558] Processing DeleteTablet for tablet 92ceada3bdd8459dafccbc3eda0ed7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:48 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:46054
I20260319 16:36:48.097275 10020 tablet_service.cc:1558] Processing DeleteTablet for tablet 92ceada3bdd8459dafccbc3eda0ed7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:48 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:36326
I20260319 16:36:48.097370 10126 tablet_replica.cc:333] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: stopping tablet replica
I20260319 16:36:48.097438 10126 raft_consensus.cc:2243] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:36:48.097465 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 9684
I20260319 16:36:48.097972 10126 raft_consensus.cc:2272] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:48.098115 10127 tablet_replica.cc:333] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: stopping tablet replica
I20260319 16:36:48.098176 10127 raft_consensus.cc:2243] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:36:48.098244 10127 raft_consensus.cc:2272] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:36:48.098706 10127 ts_tablet_manager.cc:1916] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:48.100181 10127 ts_tablet_manager.cc:1929] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:48.100260 10127 log.cc:1199] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/92ceada3bdd8459dafccbc3eda0ed7e2
I20260319 16:36:48.100310 10126 ts_tablet_manager.cc:1916] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:36:48.100558 10127 ts_tablet_manager.cc:1950] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 28fac62a7cd24d058fb1364cc38476cb: Deleting consensus metadata
I20260319 16:36:48.100878 9607 catalog_manager.cc:5002] TS 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983): tablet 92ceada3bdd8459dafccbc3eda0ed7e2 (table test-table [id=29a7dcd4767c409c84d92e83cec73473]) successfully deleted
I20260319 16:36:48.101876 10126 ts_tablet_manager.cc:1929] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:36:48.102011 10126 log.cc:1199] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/92ceada3bdd8459dafccbc3eda0ed7e2
I20260319 16:36:48.102344 10126 ts_tablet_manager.cc:1950] T 92ceada3bdd8459dafccbc3eda0ed7e2 P 45b0139f4a5e4649b1988c454b5f4fe7: Deleting consensus metadata
I20260319 16:36:48.102757 9604 catalog_manager.cc:5002] TS 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141): tablet 92ceada3bdd8459dafccbc3eda0ed7e2 (table test-table [id=29a7dcd4767c409c84d92e83cec73473]) successfully deleted
W20260319 16:36:48.104935 9605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:44517 (error 108)
I20260319 16:36:48.105129 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 9820
I20260319 16:36:48.105856 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:36:48.105311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:44517 (local address 127.0.0.1:58910)
0319 16:36:48.105784 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0319 16:36:48.105821 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453}
W20260319 16:36:48.107250 9605 catalog_manager.cc:4729] TS edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92ceada3bdd8459dafccbc3eda0ed7e2: Network error: Client connection negotiation failed: client connection to 127.24.187.193:44517: connect: Connection refused (error 111)
I20260319 16:36:48.109792 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 9957
I20260319 16:36:48.115142 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 9588
I20260319 16:37:08.120270 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35859
--webserver_interface=127.24.187.254
--webserver_port=33593
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:08.198029 10129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:08.198272 10129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:08.198313 10129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:08.199905 10129 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260319 16:37:08.199947 10129 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260319 16:37:08.199965 10129 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260319 16:37:08.199977 10129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:08.199993 10129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:08.200011 10129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:08.200034 10129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:08.200052 10129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:08.201776 10129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=33593
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:08.201995 10129 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:08.202158 10129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:37:08.208600 10129 server_base.cc:1060] running on GCE node
W20260319 16:37:08.208662 10138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:08.208726 10135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:08.208806 10136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:08.209100 10129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:08.209335 10129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:08.210515 10129 hybrid_clock.cc:648] HybridClock initialized: now 1773938228210496 us; error 32 us; skew 500 ppm
Mar 19 16:37:08 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:08.212952 10129 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:37:08.213533 10129 webserver.cc:492] Webserver started at http://127.24.187.254:33593/ using document root <none> and password file <none>
I20260319 16:37:08.213738 10129 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:08.213793 10129 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:08.215039 10129 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:08.215698 10145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.215898 10129 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:08.215961 10129 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2c657bb394ec4c4e80183c288ca36675"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "0e2561e04b199aafe1798543569ff613"
server_key_iv: "0120a97ab9409584b1b79d9b4de5154d"
server_key_version: "encryptionkey@0"
I20260319 16:37:08.216267 10129 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:08.230020 10129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:08.231971 10129 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:08.232093 10129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:08.236119 10129 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:35859
I20260319 16:37:08.236160 10198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:35859 every 8 connection(s)
I20260319 16:37:08.236470 10129 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:08.237560 10199 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:37:08.238154 10199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Bootstrap starting.
I20260319 16:37:08.245702 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 10129
I20260319 16:37:08.246098 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:44517
--local_ip_for_outbound_sockets=127.24.187.193
--tserver_master_addrs=127.24.187.254:35859
--webserver_port=46055
--webserver_interface=127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:37:08.246901 10199 log.cc:826] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:08.248757 10199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:37:08.249010 10199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Bootstrap complete.
I20260319 16:37:08.250943 10199 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:08.251124 10199 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c657bb394ec4c4e80183c288ca36675, State: Initialized, Role: FOLLOWER
I20260319 16:37:08.251248 10199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:08.251309 10199 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:08.251340 10199 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:08.251384 10199 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 1 FOLLOWER]: Advancing to term 2
I20260319 16:37:08.252197 10199 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:08.252291 10199 leader_election.cc:304] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c657bb394ec4c4e80183c288ca36675; no voters:
I20260319 16:37:08.252453 10199 leader_election.cc:290] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260319 16:37:08.252499 10203 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 FOLLOWER]: Leader election won for term 2
I20260319 16:37:08.252677 10203 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 LEADER]: Becoming Leader. State: Replica: 2c657bb394ec4c4e80183c288ca36675, State: Running, Role: LEADER
I20260319 16:37:08.252688 10199 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:08.252844 10203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:08.253192 10205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c657bb394ec4c4e80183c288ca36675. Latest consensus state: current_term: 2 leader_uuid: "2c657bb394ec4c4e80183c288ca36675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } } }
I20260319 16:37:08.253269 10205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:08.253432 10204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c657bb394ec4c4e80183c288ca36675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } } }
I20260319 16:37:08.253517 10204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:08.253691 10208 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:08.254408 10208 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:08.254508 10208 catalog_manager.cc:1269] Loaded cluster ID: 3979d3504cba4fd793224eb13bbd3c81
I20260319 16:37:08.254585 10208 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:37:08.255985 10208 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:08.256165 10208 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Loaded TSK: 0
I20260319 16:37:08.266460 10208 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Generated new TSK 1
I20260319 16:37:08.266553 10208 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260319 16:37:08.322127 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:08.322331 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:08.322371 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:08.323900 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:08.323951 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:08.323989 10202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:08.325812 10202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:44517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=46055
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:08.326182 10202 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:08.326401 10202 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:08.329038 10229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:08.329188 10202 server_base.cc:1060] running on GCE node
W20260319 16:37:08.329041 10227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:08.329038 10226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:08.329555 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:08.329836 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:08.331000 10202 hybrid_clock.cc:648] HybridClock initialized: now 1773938228330980 us; error 32 us; skew 500 ppm
Mar 19 16:37:08 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:08.333550 10202 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:37:08.334244 10202 webserver.cc:492] Webserver started at http://127.24.187.193:46055/ using document root <none> and password file <none>
I20260319 16:37:08.334458 10202 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:08.334527 10202 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:08.336148 10202 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:08.336875 10236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.337056 10202 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:37:08.337133 10202 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "edb0dfbc6ff54005ae3f1c0625649274"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "297f8cbcdd96b0ae9e6db8779b766c4e"
server_key_iv: "e7cb50bcd0aa7c04c6ad929ae51e8dcb"
server_key_version: "encryptionkey@0"
I20260319 16:37:08.337471 10202 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:08.358866 10202 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:08.361044 10202 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:08.361188 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:08.361490 10202 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:08.362226 10243 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260319 16:37:08.363301 10202 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260319 16:37:08.363375 10202 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260319 16:37:08.363451 10202 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260319 16:37:08.364179 10202 ts_tablet_manager.cc:616] Registered 1 tablets
I20260319 16:37:08.364223 10202 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260319 16:37:08.364270 10243 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap starting.
I20260319 16:37:08.372160 10202 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:44517
I20260319 16:37:08.372219 10351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:44517 every 8 connection(s)
I20260319 16:37:08.372658 10202 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:37:08.375272 10243 log.cc:826] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:08.377306 10243 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:37:08.377578 10243 tablet_bootstrap.cc:492] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap complete.
I20260319 16:37:08.378461 10243 ts_tablet_manager.cc:1403] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260319 16:37:08.379329 10243 raft_consensus.cc:359] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:37:08.379623 10243 raft_consensus.cc:740] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb0dfbc6ff54005ae3f1c0625649274, State: Initialized, Role: FOLLOWER
I20260319 16:37:08.379781 10243 consensus_queue.cc:260] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } } peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } }
I20260319 16:37:08.380057 10243 ts_tablet_manager.cc:1434] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Mar 19 16:37:08 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:08.381246 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 10202
I20260319 16:37:08.381559 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:33141
--local_ip_for_outbound_sockets=127.24.187.194
--tserver_master_addrs=127.24.187.254:35859
--webserver_port=34891
--webserver_interface=127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:37:08.384438 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.373434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:46725)
0319 16:37:08.373635 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.373672 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.374491 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.375008 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.375013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.375425 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:08.375810 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.375816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.377192 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.377193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.377857 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.377860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.378306 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.378862 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:08.378882 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:08.380516 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:08.383021 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:08.383026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:08.383028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:08.383283 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:08.383523 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:08.383524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:08.383524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:08.383586 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:08.383874 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:08.383879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:08.383986 (+ 107us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.384073 (+ 87us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.384227 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260319 16:37:08.384719 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.373828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:46725 (local address 127.24.187.254:35859)
0319 16:37:08.374019 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.374030 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.374059 (+ 29us) server_negotiation.cc:408] Connection header received
0319 16:37:08.374780 (+ 721us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.374787 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.374839 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.374976 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:08.375927 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.377088 (+ 1161us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.377959 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.378180 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.380624 (+ 2444us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:08.380644 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:08.380644 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:08.380663 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:08.382924 (+ 2261us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.383375 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.383379 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.383379 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.383446 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.383645 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.383646 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.383646 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.383813 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:08.383892 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.384151 (+ 259us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.384275 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260319 16:37:08.385492 10352 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35859
I20260319 16:37:08.385593 10352 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:08.385833 10352 heartbeater.cc:507] Master 127.24.187.254:35859 requested a full tablet report, sending...
I20260319 16:37:08.386452 10162 ts_manager.cc:194] Registered new tserver with Master: edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517)
I20260319 16:37:08.387421 10162 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:46725
I20260319 16:37:08.388082 10352 heartbeater.cc:499] Master 127.24.187.254:35859 was elected leader, sending a full tablet report...
I20260319 16:37:08.397475 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.386942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:44517 (local address 127.0.0.1:33370)
0319 16:37:08.393790 (+ 6848us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.393820 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.393898 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.395386 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.395390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.395393 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:08.395664 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.395667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.396550 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.396551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.397177 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.397178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.397252 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.397290 (+ 38us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.397339 (+ 49us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.397384 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6816,"thread_start_us":80,"threads_started":1}
I20260319 16:37:08.398021 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.387637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.24.187.193:44517)
0319 16:37:08.395160 (+ 7523us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.395172 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.395183 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:37:08.395212 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.395214 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.395226 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.395359 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:08.395747 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.396447 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.397295 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.397840 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.397886 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.397918 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.397963 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7491,"thread_start_us":85,"threads_started":1}
I20260319 16:37:08.398298 10285 tablet_service.cc:1558] Processing DeleteTablet for tablet 92ceada3bdd8459dafccbc3eda0ed7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:36:48 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33370
I20260319 16:37:08.398756 10361 tablet_replica.cc:333] stopping tablet replica
I20260319 16:37:08.398849 10361 raft_consensus.cc:2243] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:08.398921 10361 raft_consensus.cc:2272] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:08.399250 10361 ts_tablet_manager.cc:1916] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:08.401201 10361 ts_tablet_manager.cc:1929] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:08.401289 10361 log.cc:1199] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal/wals/92ceada3bdd8459dafccbc3eda0ed7e2
I20260319 16:37:08.401605 10361 ts_tablet_manager.cc:1950] T 92ceada3bdd8459dafccbc3eda0ed7e2 P edb0dfbc6ff54005ae3f1c0625649274: Deleting consensus metadata
I20260319 16:37:08.402086 10147 catalog_manager.cc:5002] TS edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517): tablet 92ceada3bdd8459dafccbc3eda0ed7e2 (table test-table [id=29a7dcd4767c409c84d92e83cec73473]) successfully deleted
W20260319 16:37:08.471985 10358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:08.472159 10358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:08.472180 10358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:08.473682 10358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:08.473742 10358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:08.473766 10358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:08.475386 10358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:33141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=34891
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:08.475550 10358 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:08.475687 10358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:08.477872 10366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:08.477981 10367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:08.478024 10358 server_base.cc:1060] running on GCE node
W20260319 16:37:08.477883 10369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:08.478260 10358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:08.478477 10358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:08.479630 10358 hybrid_clock.cc:648] HybridClock initialized: now 1773938228479615 us; error 31 us; skew 500 ppm
Mar 19 16:37:08 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:08.481928 10358 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:37:08.482383 10358 webserver.cc:492] Webserver started at http://127.24.187.194:34891/ using document root <none> and password file <none>
I20260319 16:37:08.482537 10358 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:08.482584 10358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:08.483736 10358 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:08.484205 10376 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.484408 10358 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.484464 10358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "45b0139f4a5e4649b1988c454b5f4fe7"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "fe4268c1f7cac0d4aab3d9b889c786ce"
server_key_iv: "34bc380cf34f475832ea404d79915daf"
server_key_version: "encryptionkey@0"
I20260319 16:37:08.484724 10358 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:08.491290 10358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:08.492842 10358 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:08.492952 10358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:08.493134 10358 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:08.493384 10358 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:08.493414 10358 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.493440 10358 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:08.493464 10358 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.498842 10358 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:33141
I20260319 16:37:08.498903 10489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:33141 every 8 connection(s)
I20260319 16:37:08.499141 10358 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:37:08 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:08.506763 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 10358
I20260319 16:37:08.506821 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.499764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:36229)
0319 16:37:08.499913 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.499943 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.500409 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.500585 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.500588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.500854 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:08.501123 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.501127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.501781 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.501782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.502149 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.502151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.502243 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.502729 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:08.502744 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:08.504529 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:08.505874 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:08.505879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:08.505881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:08.506063 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:08.506179 (+ 116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:08.506179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:08.506180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:08.506224 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:08.506381 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:08.506385 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:08.506481 (+ 96us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.506584 (+ 103us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.506682 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260319 16:37:08.506826 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.499806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:36229 (local address 127.24.187.254:35859)
0319 16:37:08.499859 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.499861 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.500012 (+ 151us) server_negotiation.cc:408] Connection header received
0319 16:37:08.500479 (+ 467us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.500480 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.500492 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.500544 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:08.501192 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.501679 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.502219 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.502337 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.504620 (+ 2283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:08.504625 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:08.504625 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:08.504632 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:08.505791 (+ 1159us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.506120 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.506121 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.506121 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.506138 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.506261 (+ 123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.506261 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.506261 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.506337 (+ 76us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:08.506378 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.506641 (+ 263us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.506737 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:37:08.507035 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:41983
--local_ip_for_outbound_sockets=127.24.187.195
--tserver_master_addrs=127.24.187.254:35859
--webserver_port=35507
--webserver_interface=127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:37:08.507606 10490 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35859
I20260319 16:37:08.507696 10490 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:08.507884 10490 heartbeater.cc:507] Master 127.24.187.254:35859 requested a full tablet report, sending...
I20260319 16:37:08.508217 10162 ts_manager.cc:194] Registered new tserver with Master: 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141)
I20260319 16:37:08.508692 10162 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:36229
W20260319 16:37:08.581629 10493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:08.581843 10493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:08.581868 10493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:08.583308 10493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:08.583349 10493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:08.583374 10493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:08.584923 10493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:41983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=35507
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:08.585088 10493 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:08.585213 10493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:08.587539 10501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:08.587574 10499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:08.587548 10498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:08.587746 10493 server_base.cc:1060] running on GCE node
I20260319 16:37:08.587965 10493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:08.588172 10493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:08.589327 10493 hybrid_clock.cc:648] HybridClock initialized: now 1773938228589289 us; error 43 us; skew 500 ppm
Mar 19 16:37:08 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:08.591671 10493 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:37:08.592162 10493 webserver.cc:492] Webserver started at http://127.24.187.195:35507/ using document root <none> and password file <none>
I20260319 16:37:08.592314 10493 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:08.592355 10493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:08.593465 10493 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:08.593999 10508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.594148 10493 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:37:08.594213 10493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "28fac62a7cd24d058fb1364cc38476cb"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "819f722c6c60daa2bfc81d57883ecffe"
server_key_iv: "0c31155cf243b49868c9ae733fa686c2"
server_key_version: "encryptionkey@0"
I20260319 16:37:08.594506 10493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:08.602583 10493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:08.604259 10493 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:08.604367 10493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:08.604558 10493 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:08.604846 10493 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:08.604875 10493 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.604897 10493 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:08.604919 10493 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.610385 10493 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:41983
I20260319 16:37:08.610451 10621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:41983 every 8 connection(s)
I20260319 16:37:08.610708 10493 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:08.611752 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 10493
Mar 19 16:37:08 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:08.618332 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.612122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.24.187.254:35859)
0319 16:37:08.612902 (+ 780us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.612904 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.613038 (+ 134us) server_negotiation.cc:408] Connection header received
0319 16:37:08.613075 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.613076 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.613087 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.613155 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:08.613904 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.614413 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.614965 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.615125 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.616066 (+ 941us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:08.616070 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:08.616070 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:08.616076 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:08.617458 (+ 1382us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.617758 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.617759 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.617759 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.617780 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.617892 (+ 112us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.617893 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.617893 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.617995 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:08.618110 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.618200 (+ 90us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.618262 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":60,"threads_started":1}
I20260319 16:37:08.619189 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.611406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:55239 (local address 127.24.187.254:35859)
0319 16:37:08.611460 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.611462 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.611555 (+ 93us) server_negotiation.cc:408] Connection header received
0319 16:37:08.612088 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.612089 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.612103 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.612162 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:08.612867 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.613441 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.614018 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.614177 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.616671 (+ 2494us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:08.616674 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:08.616674 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:08.616682 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:08.617956 (+ 1274us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.618366 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.618367 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.618375 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.618397 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.618618 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.618619 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.618619 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.618713 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:08.618784 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.619036 (+ 252us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.619110 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:37:08.619335 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.611271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:55239)
0319 16:37:08.611417 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.611449 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.611967 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.612227 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.612231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.612515 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:08.612785 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.612788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.613518 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.613520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.613928 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.613930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.614047 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.614549 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:08.614566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:08.616596 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:08.618013 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:08.618018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:08.618019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:08.618289 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:08.618479 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:08.618480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:08.618481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:08.618560 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:08.618775 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:08.618780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:08.618903 (+ 123us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.619011 (+ 108us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.619147 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260319 16:37:08.619721 10622 heartbeater.cc:344] Connected to a master server at 127.24.187.254:35859
I20260319 16:37:08.619804 10622 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:08.619976 10622 heartbeater.cc:507] Master 127.24.187.254:35859 requested a full tablet report, sending...
I20260319 16:37:08.620483 10162 ts_manager.cc:194] Registered new tserver with Master: 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983)
I20260319 16:37:08.620836 10162 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:55239
I20260319 16:37:08.630045 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:37:08.632391 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.630414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.24.187.254:35859)
0319 16:37:08.630477 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.630479 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.630567 (+ 88us) server_negotiation.cc:408] Connection header received
0319 16:37:08.630632 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.630633 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.630654 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.630706 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:08.630959 (+ 253us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.631432 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.632009 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.632134 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.632173 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:08.632220 (+ 47us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0319 16:37:08.632338 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56894: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260319 16:37:08.632452 10355 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56894: token expired
W20260319 16:37:08.632575 10092 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.187.254:35859)
I20260319 16:37:08.637423 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.632781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.24.187.254:35859)
0319 16:37:08.632826 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.632828 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.632835 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:37:08.632854 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.632855 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.632864 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.632915 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:08.633474 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.633941 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.634430 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.634579 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.635369 (+ 790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:08.635372 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:08.635372 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:08.635378 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:08.636541 (+ 1163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.636826 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.636827 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.636827 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.636847 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:08.636983 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:08.636984 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:08.636984 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:08.637078 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:08.637164 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.637254 (+ 90us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.637338 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:37:08.638516 10162 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:37:08.639688 10162 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:37:08.641657 10285 tablet_service.cc:1511] Processing CreateTablet for tablet 76ec6bf1b70949f394976abab020729a (DEFAULT_TABLE table=test-table [id=1724f7c926a745839e2a2d7ec670ca17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:08.641942 10285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ec6bf1b70949f394976abab020729a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:08.643253 10243 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap starting.
I20260319 16:37:08.643651 10243 tablet_bootstrap.cc:654] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:08.644126 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.641467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:41983 (local address 127.0.0.1:40088)
0319 16:37:08.641625 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.641634 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.641687 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.641949 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.641950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.641953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:08.642137 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.642140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.643147 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.643148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.643926 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.643929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.643993 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.643997 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.644025 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.644057 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260319 16:37:08.644157 10243 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: No bootstrap required, opened a new log
I20260319 16:37:08.644200 10243 ts_tablet_manager.cc:1403] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260319 16:37:08.644304 10243 raft_consensus.cc:359] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.644374 10243 raft_consensus.cc:385] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:08.644408 10243 raft_consensus.cc:740] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb0dfbc6ff54005ae3f1c0625649274, State: Initialized, Role: FOLLOWER
I20260319 16:37:08.644452 10243 consensus_queue.cc:260] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.644546 10243 ts_tablet_manager.cc:1434] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:08.644713 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.641546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.24.187.195:41983)
0319 16:37:08.641819 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.641826 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.641834 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:37:08.641863 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.641864 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.641876 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.641935 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:08.642238 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.643066 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.644011 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.644518 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.644571 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.644603 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.644644 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260319 16:37:08.645498 10556 tablet_service.cc:1511] Processing CreateTablet for tablet 76ec6bf1b70949f394976abab020729a (DEFAULT_TABLE table=test-table [id=1724f7c926a745839e2a2d7ec670ca17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:08.645694 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.641356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:33141 (local address 127.0.0.1:54738)
0319 16:37:08.641493 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.641504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.641563 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.643475 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.643476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.643479 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:08.643657 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.643660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.644849 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.644850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.645526 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.645529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.645584 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.645587 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.645607 (+ 20us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.645635 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260319 16:37:08.645778 10556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ec6bf1b70949f394976abab020729a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:08.646332 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.641670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.24.187.194:33141)
0319 16:37:08.643318 (+ 1648us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.643325 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.643334 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:08.643372 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.643374 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.643389 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.643458 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:08.643752 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.644780 (+ 1028us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.645624 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.646143 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.646186 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.646215 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.646253 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1614,"thread_start_us":105,"threads_started":1}
I20260319 16:37:08.646873 10424 tablet_service.cc:1511] Processing CreateTablet for tablet 76ec6bf1b70949f394976abab020729a (DEFAULT_TABLE table=test-table [id=1724f7c926a745839e2a2d7ec670ca17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:08.647125 10424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ec6bf1b70949f394976abab020729a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:08.648654 10632 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Bootstrap starting.
I20260319 16:37:08.649117 10632 tablet_bootstrap.cc:654] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:08.649355 10632 log.cc:826] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:08.649660 10634 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Bootstrap starting.
I20260319 16:37:08.650015 10632 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: No bootstrap required, opened a new log
I20260319 16:37:08.650079 10632 ts_tablet_manager.cc:1403] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:08.650224 10634 tablet_bootstrap.cc:654] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:08.650465 10634 log.cc:826] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:08.651096 10634 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: No bootstrap required, opened a new log
I20260319 16:37:08.651145 10634 ts_tablet_manager.cc:1403] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260319 16:37:08.651337 10632 raft_consensus.cc:359] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.651448 10632 raft_consensus.cc:385] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:08.651473 10632 raft_consensus.cc:740] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fac62a7cd24d058fb1364cc38476cb, State: Initialized, Role: FOLLOWER
I20260319 16:37:08.651578 10632 consensus_queue.cc:260] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.651778 10632 ts_tablet_manager.cc:1434] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:08.651825 10622 heartbeater.cc:499] Master 127.24.187.254:35859 was elected leader, sending a full tablet report...
I20260319 16:37:08.652344 10634 raft_consensus.cc:359] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.652469 10634 raft_consensus.cc:385] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:08.652509 10634 raft_consensus.cc:740] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b0139f4a5e4649b1988c454b5f4fe7, State: Initialized, Role: FOLLOWER
I20260319 16:37:08.652590 10634 consensus_queue.cc:260] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.652801 10634 ts_tablet_manager.cc:1434] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:37:08.652843 10490 heartbeater.cc:499] Master 127.24.187.254:35859 was elected leader, sending a full tablet report...
W20260319 16:37:08.749859 10491 tablet.cc:2379] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:08.852533 10637 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:08.852670 10637 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.853022 10637 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983), edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517)
I20260319 16:37:08.853269 10636 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:08.853453 10636 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.853870 10636 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141), edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517)
I20260319 16:37:08.855818 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.853198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:41983 (local address 127.24.187.194:37171)
0319 16:37:08.853334 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.853346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.853400 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.853634 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.853635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.853637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:08.853881 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.853884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.854790 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.854791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.855635 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.855637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.855689 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.855693 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.855721 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.855752 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260319 16:37:08.856575 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.854042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:33141 (local address 127.24.187.195:43279)
0319 16:37:08.854124 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.854133 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.854164 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.854508 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.854509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.854511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:08.854694 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.854695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.855555 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.855556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.856366 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.856368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.856441 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.856446 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.856476 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.856506 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260319 16:37:08.856607 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.853393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:47613 (local address 127.24.187.193:44517)
0319 16:37:08.853437 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.853439 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.853505 (+ 66us) server_negotiation.cc:408] Connection header received
0319 16:37:08.853556 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.853561 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.853569 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.853619 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:08.854172 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.854897 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.855947 (+ 1050us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.856391 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.856399 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.856506 (+ 107us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.856541 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260319 16:37:08.856727 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.854103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:44517 (local address 127.24.187.195:39855)
0319 16:37:08.854273 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.854281 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.854326 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.854536 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.854537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.854539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:08.854830 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.854831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.855684 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.855685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.856530 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.856533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.856597 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.856600 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.856627 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.856660 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260319 16:37:08.856793 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.853208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:44517 (local address 127.24.187.194:47613)
0319 16:37:08.853407 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.853418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:08.853501 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:08.853634 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:08.853635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:08.853637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:08.854039 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.854042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.854979 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.854980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:08.855864 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:08.855867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.856427 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.856430 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:08.856678 (+ 248us) client_negotiation.cc:240] Negotiation successful
0319 16:37:08.856716 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":11,"thread_start_us":106,"threads_started":1}
I20260319 16:37:08.857169 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.854220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:43279 (local address 127.24.187.194:33141)
0319 16:37:08.854397 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.854399 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.854405 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:37:08.854422 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.854423 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.854430 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.854470 (+ 40us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:08.854778 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.855472 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.856623 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.857065 (+ 442us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.857072 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.857090 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.857117 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260319 16:37:08.857357 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.854298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:39855 (local address 127.24.187.193:44517)
0319 16:37:08.854390 (+ 92us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.854392 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.854405 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:37:08.854425 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.854426 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.854435 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.854488 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:08.854906 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.855601 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.856873 (+ 1272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.857202 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.857207 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.857225 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.857253 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
I20260319 16:37:08.857363 10444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" is_pre_election: true
I20260319 16:37:08.857409 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.853261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:37171 (local address 127.24.187.195:41983)
0319 16:37:08.853313 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.853315 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.853458 (+ 143us) server_negotiation.cc:408] Connection header received
0319 16:37:08.853491 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.853492 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.853503 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.853546 (+ 43us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:08.853972 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.854720 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.856104 (+ 1384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.856485 (+ 381us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.856490 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.856507 (+ 17us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.856532 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:37:08.857434 10444 raft_consensus.cc:2468] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28fac62a7cd24d058fb1364cc38476cb in term 0.
I20260319 16:37:08.857285 10305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb0dfbc6ff54005ae3f1c0625649274" is_pre_election: true
I20260319 16:37:08.857613 10509 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fac62a7cd24d058fb1364cc38476cb, 45b0139f4a5e4649b1988c454b5f4fe7; no voters:
I20260319 16:37:08.857640 10304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb0dfbc6ff54005ae3f1c0625649274" is_pre_election: true
I20260319 16:37:08.857695 10305 raft_consensus.cc:2468] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b0139f4a5e4649b1988c454b5f4fe7 in term 0.
I20260319 16:37:08.857746 10636 raft_consensus.cc:2804] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:08.857782 10636 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:08.857823 10636 raft_consensus.cc:3060] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:08.857613 10576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fac62a7cd24d058fb1364cc38476cb" is_pre_election: true
I20260319 16:37:08.857981 10378 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b0139f4a5e4649b1988c454b5f4fe7, edb0dfbc6ff54005ae3f1c0625649274; no voters:
I20260319 16:37:08.858093 10637 raft_consensus.cc:2804] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:08.858132 10637 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:08.858152 10637 raft_consensus.cc:3060] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:08.858521 10636 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.858654 10636 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 election: Requested vote from peers 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141), edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517)
I20260319 16:37:08.858659 10576 raft_consensus.cc:2393] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 45b0139f4a5e4649b1988c454b5f4fe7 in current term 1: Already voted for candidate 28fac62a7cd24d058fb1364cc38476cb in this term.
I20260319 16:37:08.858794 10304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb0dfbc6ff54005ae3f1c0625649274"
I20260319 16:37:08.858817 10637 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.858860 10304 raft_consensus.cc:3060] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:08.858929 10637 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [CANDIDATE]: Term 1 election: Requested vote from peers 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983), edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517)
I20260319 16:37:08.859081 10444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "28fac62a7cd24d058fb1364cc38476cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b0139f4a5e4649b1988c454b5f4fe7"
I20260319 16:37:08.859104 10305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb0dfbc6ff54005ae3f1c0625649274"
I20260319 16:37:08.859113 10576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ec6bf1b70949f394976abab020729a" candidate_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fac62a7cd24d058fb1364cc38476cb"
I20260319 16:37:08.859153 10444 raft_consensus.cc:2393] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 28fac62a7cd24d058fb1364cc38476cb in current term 1: Already voted for candidate 45b0139f4a5e4649b1988c454b5f4fe7 in this term.
I20260319 16:37:08.859167 10576 raft_consensus.cc:2393] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 45b0139f4a5e4649b1988c454b5f4fe7 in current term 1: Already voted for candidate 28fac62a7cd24d058fb1364cc38476cb in this term.
I20260319 16:37:08.859313 10378 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45b0139f4a5e4649b1988c454b5f4fe7; no voters: 28fac62a7cd24d058fb1364cc38476cb, edb0dfbc6ff54005ae3f1c0625649274
I20260319 16:37:08.859396 10637 raft_consensus.cc:2749] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260319 16:37:08.859611 10304 raft_consensus.cc:2468] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28fac62a7cd24d058fb1364cc38476cb in term 1.
I20260319 16:37:08.859771 10510 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28fac62a7cd24d058fb1364cc38476cb, edb0dfbc6ff54005ae3f1c0625649274; no voters: 45b0139f4a5e4649b1988c454b5f4fe7
I20260319 16:37:08.859866 10636 raft_consensus.cc:2804] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:08.859990 10636 raft_consensus.cc:697] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 1 LEADER]: Becoming Leader. State: Replica: 28fac62a7cd24d058fb1364cc38476cb, State: Running, Role: LEADER
I20260319 16:37:08.860080 10636 consensus_queue.cc:237] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:08.860780 10162 catalog_manager.cc:5671] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195). New cstate: current_term: 1 leader_uuid: "28fac62a7cd24d058fb1364cc38476cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } health_report { overall_health: UNKNOWN } } }
W20260319 16:37:08.861132 10623 tablet.cc:2379] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:08.871718 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.869802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.24.187.195:41983)
0319 16:37:08.869871 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:08.869873 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:08.869886 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:37:08.869919 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:08.869920 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:08.869936 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:08.869991 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:08.870248 (+ 257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.870702 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:08.871222 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:08.871369 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:08.871400 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:08.871474 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:37:08.871531 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:08.871648 (+ 117us) server_negotiation.cc:300] Negotiation successful
0319 16:37:08.871670 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260319 16:37:08.873036 10304 raft_consensus.cc:1275] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Refusing update from remote peer 28fac62a7cd24d058fb1364cc38476cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:08.873032 10444 raft_consensus.cc:1275] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Refusing update from remote peer 28fac62a7cd24d058fb1364cc38476cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:08.873328 10641 consensus_queue.cc:1048] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:08.873438 10636 consensus_queue.cc:1048] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:08.875560 10644 mvcc.cc:204] Tried to move back new op lower bound from 7266050985461899264 to 7266050985411338240. Current Snapshot: MvccSnapshot[applied={T|T < 7266050985461899264}]
I20260319 16:37:08.875913 10645 mvcc.cc:204] Tried to move back new op lower bound from 7266050985461899264 to 7266050985411338240. Current Snapshot: MvccSnapshot[applied={T|T < 7266050985461899264}]
I20260319 16:37:08.877568 10162 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:08.877648 10162 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:08.878341 10162 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Sending DeleteTablet for 3 replicas of tablet 76ec6bf1b70949f394976abab020729a
I20260319 16:37:08.878615 10285 tablet_service.cc:1558] Processing DeleteTablet for tablet 76ec6bf1b70949f394976abab020729a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:08 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:33370
I20260319 16:37:08.878609 10556 tablet_service.cc:1558] Processing DeleteTablet for tablet 76ec6bf1b70949f394976abab020729a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:08 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:40088
I20260319 16:37:08.878650 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 10202
I20260319 16:37:08.878720 10361 tablet_replica.cc:333] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: stopping tablet replica
I20260319 16:37:08.878764 10361 raft_consensus.cc:2243] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:08.878798 10361 raft_consensus.cc:2272] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:08.879208 10424 tablet_service.cc:1558] Processing DeleteTablet for tablet 76ec6bf1b70949f394976abab020729a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:08 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:54738
I20260319 16:37:08.879411 10654 tablet_replica.cc:333] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: stopping tablet replica
I20260319 16:37:08.879487 10654 raft_consensus.cc:2243] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:08.879581 10654 raft_consensus.cc:2272] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:08.879676 10653 tablet_replica.cc:333] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: stopping tablet replica
I20260319 16:37:08.879745 10653 raft_consensus.cc:2243] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:37:08.879818 10653 raft_consensus.cc:2272] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:08.879976 10654 ts_tablet_manager.cc:1916] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:08.880191 10653 ts_tablet_manager.cc:1916] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:08.881623 10653 ts_tablet_manager.cc:1929] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:08.881628 10654 ts_tablet_manager.cc:1929] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:08.881690 10654 log.cc:1199] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/76ec6bf1b70949f394976abab020729a
I20260319 16:37:08.881690 10653 log.cc:1199] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/76ec6bf1b70949f394976abab020729a
I20260319 16:37:08.881968 10654 ts_tablet_manager.cc:1950] T 76ec6bf1b70949f394976abab020729a P 45b0139f4a5e4649b1988c454b5f4fe7: Deleting consensus metadata
I20260319 16:37:08.881989 10653 ts_tablet_manager.cc:1950] T 76ec6bf1b70949f394976abab020729a P 28fac62a7cd24d058fb1364cc38476cb: Deleting consensus metadata
I20260319 16:37:08.882316 10146 catalog_manager.cc:5002] TS 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141): tablet 76ec6bf1b70949f394976abab020729a (table test-table [id=1724f7c926a745839e2a2d7ec670ca17]) successfully deleted
I20260319 16:37:08.882460 10149 catalog_manager.cc:5002] TS 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983): tablet 76ec6bf1b70949f394976abab020729a (table test-table [id=1724f7c926a745839e2a2d7ec670ca17]) successfully deleted
W20260319 16:37:08.884028 10147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:44517 (error 108)
I20260319 16:37:08.884428 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:08.884331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:44517 (local address 127.0.0.1:33378)
0319 16:37:08.884374 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:08.884402 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260319 16:37:08.884524 10147 catalog_manager.cc:4729] TS edb0dfbc6ff54005ae3f1c0625649274 (127.24.187.193:44517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76ec6bf1b70949f394976abab020729a: Network error: Client connection negotiation failed: client connection to 127.24.187.193:44517: connect: Connection refused (error 111)
I20260319 16:37:08.885206 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 10358
I20260319 16:37:08.890095 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 10493
I20260319 16:37:08.895171 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 10129
I20260319 16:37:28.900274 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35859
--webserver_interface=127.24.187.254
--webserver_port=33593
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:28.975018 10656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:28.975219 10656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:28.975242 10656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:28.976801 10656 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260319 16:37:28.976848 10656 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260319 16:37:28.976861 10656 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260319 16:37:28.976871 10656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:28.976881 10656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:28.976893 10656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:28.976903 10656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:28.976917 10656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:28.978621 10656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=33593
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:28.978823 10656 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:28.978998 10656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:28.981202 10664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:28.981220 10662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:28.981426 10656 server_base.cc:1060] running on GCE node
W20260319 16:37:28.981210 10661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:28.981659 10656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:28.981900 10656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:28.983043 10656 hybrid_clock.cc:648] HybridClock initialized: now 1773938248983028 us; error 27 us; skew 500 ppm
Mar 19 16:37:28 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938248, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:28.985388 10656 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:37:28.985886 10656 webserver.cc:492] Webserver started at http://127.24.187.254:33593/ using document root <none> and password file <none>
I20260319 16:37:28.986032 10656 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:28.986063 10656 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:28.987151 10656 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:28.987738 10671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:28.987938 10656 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:37:28.988003 10656 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2c657bb394ec4c4e80183c288ca36675"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "0e2561e04b199aafe1798543569ff613"
server_key_iv: "0120a97ab9409584b1b79d9b4de5154d"
server_key_version: "encryptionkey@0"
I20260319 16:37:28.988286 10656 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:29.005788 10656 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:29.007665 10656 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:29.007786 10656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:29.011637 10656 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:35859
I20260319 16:37:29.011669 10723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:35859 every 8 connection(s)
I20260319 16:37:29.011965 10656 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:29.013063 10724 sys_catalog.cc:263] Verifying existing consensus state
I20260319 16:37:29.013618 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Bootstrap starting.
I20260319 16:37:29.015070 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 10656
I20260319 16:37:29.015345 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:44517
--local_ip_for_outbound_sockets=127.24.187.193
--tserver_master_addrs=127.24.187.254:35859
--webserver_port=46055
--webserver_interface=127.24.187.193
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:37:29.022426 10724 log.cc:826] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:29.024560 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:37:29.024825 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: Bootstrap complete.
I20260319 16:37:29.026736 10724 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:29.026925 10724 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c657bb394ec4c4e80183c288ca36675, State: Initialized, Role: FOLLOWER
I20260319 16:37:29.027062 10724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:29.027135 10724 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:29.027179 10724 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:29.027223 10724 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 2 FOLLOWER]: Advancing to term 3
I20260319 16:37:29.028018 10724 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:29.028115 10724 leader_election.cc:304] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c657bb394ec4c4e80183c288ca36675; no voters:
I20260319 16:37:29.028259 10724 leader_election.cc:290] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260319 16:37:29.028307 10728 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 3 FOLLOWER]: Leader election won for term 3
I20260319 16:37:29.028450 10728 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [term 3 LEADER]: Becoming Leader. State: Replica: 2c657bb394ec4c4e80183c288ca36675, State: Running, Role: LEADER
I20260319 16:37:29.028496 10724 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:29.028669 10728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } }
I20260319 16:37:29.029615 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2c657bb394ec4c4e80183c288ca36675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } } }
I20260319 16:37:29.029702 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:29.029927 10729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c657bb394ec4c4e80183c288ca36675. Latest consensus state: current_term: 3 leader_uuid: "2c657bb394ec4c4e80183c288ca36675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c657bb394ec4c4e80183c288ca36675" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35859 } } }
I20260319 16:37:29.030123 10729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:29.030274 10744 catalog_manager.cc:1269] Loaded cluster ID: 3979d3504cba4fd793224eb13bbd3c81
I20260319 16:37:29.030320 10744 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: loading cluster ID for follower catalog manager: success
I20260319 16:37:29.031617 10744 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 2c657bb394ec4c4e80183c288ca36675: acquiring CA information for follower catalog manager: success
F20260319 16:37:29.031967 10744 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fca28982dcd google::LogMessage::Fail() at ??:0
@ 0x7fca28986b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fca289827cc google::LogMessage::Flush() at ??:0
@ 0x7fca28983f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fca271b7256 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fca2b1fbfc5 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fca2b1fc268 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fca2b1fc7d3 kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fca2b1fed27 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fca29ab51b7 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fca29dac6db start_thread at ??:0
@ 0x7fca27ed761f clone at ??:0
W20260319 16:37:29.105206 10726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:29.105372 10726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:29.105396 10726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:29.106938 10726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:29.106985 10726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:29.107010 10726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:29.108654 10726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:44517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=46055
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:29.108857 10726 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:29.109027 10726 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:29.111354 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:29.111404 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:29.111363 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:29.111546 10726 server_base.cc:1060] running on GCE node
I20260319 16:37:29.111727 10726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:29.111937 10726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:29.113097 10726 hybrid_clock.cc:648] HybridClock initialized: now 1773938249113077 us; error 31 us; skew 500 ppm
Mar 19 16:37:29 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:29.115906 10726 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:37:29.116533 10726 webserver.cc:492] Webserver started at http://127.24.187.193:46055/ using document root <none> and password file <none>
I20260319 16:37:29.116729 10726 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:29.116775 10726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:29.118294 10726 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:29.118994 10773 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.119169 10726 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:29.119241 10726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "edb0dfbc6ff54005ae3f1c0625649274"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "297f8cbcdd96b0ae9e6db8779b766c4e"
server_key_iv: "e7cb50bcd0aa7c04c6ad929ae51e8dcb"
server_key_version: "encryptionkey@0"
I20260319 16:37:29.119588 10726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:29.137799 10726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:29.139866 10726 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:29.140007 10726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:29.140271 10726 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:29.140743 10780 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260319 16:37:29.141615 10726 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260319 16:37:29.141670 10726 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260319 16:37:29.141702 10726 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260319 16:37:29.142431 10726 ts_tablet_manager.cc:616] Registered 1 tablets
I20260319 16:37:29.142475 10726 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260319 16:37:29.142534 10780 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap starting.
I20260319 16:37:29.149968 10726 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:44517
I20260319 16:37:29.150149 10888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:44517 every 8 connection(s)
I20260319 16:37:29.150429 10726 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:37:29.152539 10780 log.cc:826] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:29.154870 10780 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260319 16:37:29.155172 10780 tablet_bootstrap.cc:492] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Bootstrap complete.
I20260319 16:37:29.156226 10780 ts_tablet_manager.cc:1403] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260319 16:37:29.157377 10780 raft_consensus.cc:359] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:29.157648 10780 raft_consensus.cc:740] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb0dfbc6ff54005ae3f1c0625649274, State: Initialized, Role: FOLLOWER
I20260319 16:37:29.157796 10780 consensus_queue.cc:260] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:29.158039 10780 ts_tablet_manager.cc:1434] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260319 16:37:29.160502 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 10726
I20260319 16:37:29.160769 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:33141
--local_ip_for_outbound_sockets=127.24.187.194
--tserver_master_addrs=127.24.187.254:35859
--webserver_port=34891
--webserver_interface=127.24.187.194
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:29.228497 10891 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:37:29.151107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:55455)
0319 16:37:29.151301 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.151338 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:29.151986 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:29.228332 (+ 76346us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
W20260319 16:37:29.228760 10889 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:35859 (0 consecutive failures): Network error: Failed to ping master at 127.24.187.254:35859: Client connection negotiation failed: client connection to 127.24.187.254:35859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260319 16:37:29.229099 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.228991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:43281)
0319 16:37:29.229040 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.229075 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:37:29.229420 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.229342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:52057)
0319 16:37:29.229376 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.229399 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260319 16:37:29.229542 10889 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260319 16:37:29.237344 10894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:29.237520 10894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:29.237543 10894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:29.239094 10894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:29.239141 10894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:29.239167 10894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:29.240754 10894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:33141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=34891
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:29.240914 10894 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:29.241048 10894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:29.243382 10900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:29.243448 10902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:29.243512 10894 server_base.cc:1060] running on GCE node
W20260319 16:37:29.243407 10899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:29.243749 10894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:29.243935 10894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:29.245080 10894 hybrid_clock.cc:648] HybridClock initialized: now 1773938249245057 us; error 33 us; skew 500 ppm
Mar 19 16:37:29 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:29.247407 10894 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:37:29.247896 10894 webserver.cc:492] Webserver started at http://127.24.187.194:34891/ using document root <none> and password file <none>
I20260319 16:37:29.248044 10894 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:29.248073 10894 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:29.249219 10894 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:29.249907 10909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.250074 10894 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:29.250139 10894 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "45b0139f4a5e4649b1988c454b5f4fe7"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "fe4268c1f7cac0d4aab3d9b889c786ce"
server_key_iv: "34bc380cf34f475832ea404d79915daf"
server_key_version: "encryptionkey@0"
I20260319 16:37:29.250456 10894 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:29.260277 10894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:29.262089 10894 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:29.262216 10894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:29.262403 10894 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:29.262676 10894 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:29.262707 10894 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.262734 10894 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:29.262758 10894 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.268278 10894 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:33141
I20260319 16:37:29.268362 11022 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:33141 every 8 connection(s)
I20260319 16:37:29.268604 10894 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:37:29.269572 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.269217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:50541)
0319 16:37:29.269392 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.269464 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
W20260319 16:37:29.269783 11023 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:35859 (0 consecutive failures): Network error: Failed to ping master at 127.24.187.254:35859: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
I20260319 16:37:29.270043 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.269947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:53911)
0319 16:37:29.269990 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.270019 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:37:29.270332 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.270257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:38667)
0319 16:37:29.270292 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.270315 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260319 16:37:29.270419 11023 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260319 16:37:29.275728 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 10894
I20260319 16:37:29.276016 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:41983
--local_ip_for_outbound_sockets=127.24.187.195
--tserver_master_addrs=127.24.187.254:35859
--webserver_port=35507
--webserver_interface=127.24.187.195
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:29.350780 11026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:29.350943 11026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:29.350965 11026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:29.352385 11026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:29.352423 11026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:29.352447 11026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:29.354090 11026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:41983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=35507
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:29.354246 11026 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:29.354383 11026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:29.356773 11034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:29.356742 11032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:29.356742 11031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:29.356915 11026 server_base.cc:1060] running on GCE node
I20260319 16:37:29.357152 11026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:29.357316 11026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:29.358461 11026 hybrid_clock.cc:648] HybridClock initialized: now 1773938249358444 us; error 28 us; skew 500 ppm
Mar 19 16:37:29 dist-test-slave-svn8 krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:29.360793 11026 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:37:29.361287 11026 webserver.cc:492] Webserver started at http://127.24.187.195:35507/ using document root <none> and password file <none>
I20260319 16:37:29.361434 11026 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:29.361477 11026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:29.362636 11026 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:29.363128 11041 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.363281 11026 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.363330 11026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "28fac62a7cd24d058fb1364cc38476cb"
format_stamp: "Formatted at 2026-03-19 16:36:47 on dist-test-slave-svn8"
server_key: "819f722c6c60daa2bfc81d57883ecffe"
server_key_iv: "0c31155cf243b49868c9ae733fa686c2"
server_key_version: "encryptionkey@0"
I20260319 16:37:29.363612 11026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:29.373297 11026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:29.374893 11026 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:29.374996 11026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:29.375200 11026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:29.375475 11026 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:29.375506 11026 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.375533 11026 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:29.375560 11026 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:29.381201 11026 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:41983
I20260319 16:37:29.381279 11155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:41983 every 8 connection(s)
I20260319 16:37:29.381537 11026 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:29.382469 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.382117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:59165)
0319 16:37:29.382285 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.382365 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
W20260319 16:37:29.382716 11156 heartbeater.cc:646] Failed to heartbeat to 127.24.187.254:35859 (0 consecutive failures): Network error: Failed to ping master at 127.24.187.254:35859: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
I20260319 16:37:29.383016 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.382930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:50219)
0319 16:37:29.382969 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.382996 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260319 16:37:29.383281 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:29.383214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:46731)
0319 16:37:29.383248 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:29.383268 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260319 16:37:29.383388 11156 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260319 16:37:29.390806 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11026
I20260319 16:37:30.230194 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:30.229926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:44561)
0319 16:37:30.230091 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:30.230133 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260319 16:37:30.271001 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:30.270712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:59675)
0319 16:37:30.270878 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:30.270926 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260319 16:37:30.383942 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:30.383734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:44939)
0319 16:37:30.383849 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:30.383888 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260319 16:37:30.647423 11163 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:30.647531 11163 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:30.647789 11163 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141), 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983)
Mar 19 16:37:30 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.195@KRBTEST.COM
Mar 19 16:37:30 dist-test-slave-svn8 krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.194@KRBTEST.COM
I20260319 16:37:30.662040 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:30.648055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:46271 (local address 127.24.187.194:33141)
0319 16:37:30.648190 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:30.648206 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:30.648221 (+ 15us) server_negotiation.cc:408] Connection header received
0319 16:37:30.648645 (+ 424us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:30.648647 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:30.648683 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:30.648783 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:30.649942 (+ 1159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.650931 (+ 989us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:30.651476 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.651717 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:30.658732 (+ 7015us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:30.658764 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:30.658764 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:30.658784 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:30.660476 (+ 1692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:30.660993 (+ 517us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:30.660996 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:30.660996 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:30.661063 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:30.661218 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:30.661219 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:30.661219 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:30.661444 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:30.661528 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:30.661793 (+ 265us) server_negotiation.cc:300] Negotiation successful
0319 16:37:30.661919 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260319 16:37:30.662043 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:30.647943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:41983 (local address 127.24.187.193:43027)
0319 16:37:30.647999 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:30.648010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:30.648064 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:30.648835 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:30.648836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:30.649519 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:30.649828 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:30.649833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.651078 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:30.651079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:30.651444 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:30.651446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.651516 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:30.656276 (+ 4760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:30.656307 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:30.658311 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:30.660488 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:30.660494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:30.660496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:30.660797 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:30.661053 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:30.661054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:30.661055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:30.661131 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:30.661490 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:30.661496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:30.661609 (+ 113us) client_negotiation.cc:769] Sending connection context
0319 16:37:30.661698 (+ 89us) client_negotiation.cc:240] Negotiation successful
0319 16:37:30.661950 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260319 16:37:30.662142 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:30.648049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:43027 (local address 127.24.187.195:41983)
0319 16:37:30.648176 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:30.648194 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:30.648210 (+ 16us) server_negotiation.cc:408] Connection header received
0319 16:37:30.648671 (+ 461us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:30.648674 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:30.648708 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:30.648818 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:30.649950 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.650979 (+ 1029us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:30.651522 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.651769 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:30.658449 (+ 6680us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:30.658480 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:30.658480 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:30.658514 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:30.660389 (+ 1875us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:30.660899 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:30.660903 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:30.660904 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:30.660979 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:30.661190 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:30.661191 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:30.661191 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:30.661429 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:30.661518 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:30.661804 (+ 286us) server_negotiation.cc:300] Negotiation successful
0319 16:37:30.662033 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260319 16:37:30.662173 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:30.647965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:33141 (local address 127.24.187.193:46271)
0319 16:37:30.648077 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:30.648085 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:30.648127 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:30.648833 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:30.648835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:30.649480 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:30.649827 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:30.649832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.651026 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:30.651027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:30.651398 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:30.651400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:30.651506 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:30.656276 (+ 4770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:30.656307 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:30.658629 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:30.660564 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:30.660565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:30.660566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:30.660896 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:30.661136 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:30.661136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:30.661137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:30.661165 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:30.661503 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:30.661504 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:30.661603 (+ 99us) client_negotiation.cc:769] Sending connection context
0319 16:37:30.661698 (+ 95us) client_negotiation.cc:240] Negotiation successful
0319 16:37:30.662070 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":116,"thread_start_us":50,"threads_started":1}
W20260319 16:37:30.662883 10777 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
W20260319 16:37:30.662986 10774 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
I20260319 16:37:30.663057 10777 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edb0dfbc6ff54005ae3f1c0625649274; no voters: 28fac62a7cd24d058fb1364cc38476cb, 45b0139f4a5e4649b1988c454b5f4fe7
I20260319 16:37:30.663272 11163 raft_consensus.cc:2749] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260319 16:37:31.231000 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:31.230750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:45893)
0319 16:37:31.230906 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:31.230953 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260319 16:37:31.271782 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:31.271510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:42507)
0319 16:37:31.271693 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:31.271731 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260319 16:37:31.384624 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:31.384393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:53117)
0319 16:37:31.384541 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:31.384580 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260319 16:37:32.231789 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:32.231534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:41289)
0319 16:37:32.231700 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:32.231741 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260319 16:37:32.272482 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:32.272251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:60253)
0319 16:37:32.272403 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:32.272441 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260319 16:37:32.385321 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:32.385084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:33949)
0319 16:37:32.385215 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:32.385252 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260319 16:37:32.821993 11173 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:32.822104 11173 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:32.822398 11173 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141), 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983)
W20260319 16:37:32.822711 10777 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
W20260319 16:37:32.822890 10774 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
I20260319 16:37:32.822958 10774 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edb0dfbc6ff54005ae3f1c0625649274; no voters: 28fac62a7cd24d058fb1364cc38476cb, 45b0139f4a5e4649b1988c454b5f4fe7
I20260319 16:37:32.823035 11173 raft_consensus.cc:2749] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260319 16:37:33.232550 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:33.232288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:51523)
0319 16:37:33.232455 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:33.232501 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260319 16:37:33.273254 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:33.273004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:42283)
0319 16:37:33.273148 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:33.273190 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260319 16:37:33.386073 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:33.385821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:48753)
0319 16:37:33.385984 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:33.386025 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260319 16:37:34.233323 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:34.233098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:49111)
0319 16:37:34.233239 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:34.233273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260319 16:37:34.274031 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:34.273825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:44507)
0319 16:37:34.273948 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:34.273985 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260319 16:37:34.386788 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:34.386550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:33757)
0319 16:37:34.386704 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:34.386746 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260319 16:37:35.234076 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:35.233841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:50773)
0319 16:37:35.233989 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:35.234027 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260319 16:37:35.274639 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:35.274447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:49945)
0319 16:37:35.274558 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:35.274590 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260319 16:37:35.387591 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:35.387290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:57055)
0319 16:37:35.387499 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:35.387544 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":66,"threads_started":1}
I20260319 16:37:36.185595 11183 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:36.185702 11183 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:36.185930 11183 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141), 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983)
W20260319 16:37:36.186309 10777 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
W20260319 16:37:36.186432 10774 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
I20260319 16:37:36.186491 10774 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edb0dfbc6ff54005ae3f1c0625649274; no voters: 28fac62a7cd24d058fb1364cc38476cb, 45b0139f4a5e4649b1988c454b5f4fe7
I20260319 16:37:36.186571 11183 raft_consensus.cc:2749] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260319 16:37:36.234871 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:36.234595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:45021)
0319 16:37:36.234770 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:36.234810 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260319 16:37:36.275254 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:36.275027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:55435)
0319 16:37:36.275174 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:36.275208 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260319 16:37:36.388339 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:36.388075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:38845)
0319 16:37:36.388243 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:36.388293 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260319 16:37:37.235572 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:37.235339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:37291)
0319 16:37:37.235481 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:37.235520 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260319 16:37:37.275970 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:37.275734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:58347)
0319 16:37:37.275889 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:37.275922 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260319 16:37:37.389093 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:37.388833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:44185)
0319 16:37:37.388996 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:37.389047 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260319 16:37:38.236330 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:38.236091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:54327)
0319 16:37:38.236214 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:38.236261 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260319 16:37:38.276664 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:38.276419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:49119)
0319 16:37:38.276552 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:38.276590 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260319 16:37:38.389873 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:38.389637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:32927)
0319 16:37:38.389784 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:38.389826 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260319 16:37:39.237126 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:39.236917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:45925)
0319 16:37:39.237040 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:39.237071 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260319 16:37:39.277366 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:39.277152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:49353)
0319 16:37:39.277277 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:39.277318 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260319 16:37:39.390612 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:39.390363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:42917)
0319 16:37:39.390506 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:39.390563 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260319 16:37:40.167064 11196 raft_consensus.cc:493] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:40.167153 11196 raft_consensus.cc:515] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b0139f4a5e4649b1988c454b5f4fe7" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 33141 } } peers { permanent_uuid: "28fac62a7cd24d058fb1364cc38476cb" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 41983 } } peers { permanent_uuid: "edb0dfbc6ff54005ae3f1c0625649274" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 44517 } }
I20260319 16:37:40.167301 11196 leader_election.cc:290] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141), 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983)
W20260319 16:37:40.167677 10774 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45b0139f4a5e4649b1988c454b5f4fe7 (127.24.187.194:33141): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
W20260319 16:37:40.167788 10777 leader_election.cc:343] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28fac62a7cd24d058fb1364cc38476cb (127.24.187.195:41983): Not found: Tablet not found: 76ec6bf1b70949f394976abab020729a
I20260319 16:37:40.167851 10777 leader_election.cc:304] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edb0dfbc6ff54005ae3f1c0625649274; no voters: 28fac62a7cd24d058fb1364cc38476cb, 45b0139f4a5e4649b1988c454b5f4fe7
I20260319 16:37:40.167917 11196 raft_consensus.cc:2749] T 76ec6bf1b70949f394976abab020729a P edb0dfbc6ff54005ae3f1c0625649274 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260319 16:37:40.237854 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:40.237591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:44055)
0319 16:37:40.237767 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:40.237806 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260319 16:37:40.278137 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:40.277875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:58751)
0319 16:37:40.278051 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:40.278090 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260319 16:37:40.391388 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:40.391147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:44633)
0319 16:37:40.391286 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:40.391329 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260319 16:37:41.238626 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:41.238347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:57003)
0319 16:37:41.238533 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:41.238562 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260319 16:37:41.278859 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:41.278621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:44261)
0319 16:37:41.278767 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:41.278810 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260319 16:37:41.392150 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:41.391885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:36659)
0319 16:37:41.392054 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:41.392102 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260319 16:37:42.239398 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:42.239139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:49755)
0319 16:37:42.239272 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:42.239345 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260319 16:37:42.279626 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:42.279407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:54155)
0319 16:37:42.279526 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:42.279565 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260319 16:37:42.392952 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:42.392715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:40585)
0319 16:37:42.392865 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:42.392905 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260319 16:37:43.240135 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:43.239898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:41617)
0319 16:37:43.240044 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:43.240087 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260319 16:37:43.280393 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:43.280137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:55367)
0319 16:37:43.280298 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:43.280343 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260319 16:37:43.393800 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:43.393491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:49083)
0319 16:37:43.393675 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:43.393744 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260319 16:37:44.240968 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:44.240730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.193:34487)
0319 16:37:44.240877 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:44.240913 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260319 16:37:44.281271 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:44.281027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.194:59239)
0319 16:37:44.281180 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:44.281221 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260319 16:37:44.394651 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:44.394417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35859 (local address 127.24.187.195:43231)
0319 16:37:44.394555 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:44.394592 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.254:35859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260319 16:37:44.398684 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 10726
I20260319 16:37:44.403672 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 10894
I20260319 16:37:44.408032 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11026
2026-03-19T16:37:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59054 ms)
[----------] 31 tests from SecurityITest (296523 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:37:44 dist-test-slave-svn8 krb5kdc[11214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:37:44 dist-test-slave-svn8 krb5kdc[11214](info): set up 2 sockets
Mar 19 16:37:44 dist-test-slave-svn8 krb5kdc[11214](info): commencing operation
krb5kdc: starting...
W20260319 16:37:46.258077 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.831s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:37:46 dist-test-slave-svn8 krb5kdc[11214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:37:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:46.397392 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36565
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:36565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:46.472891 11230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:46.473069 11230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:46.473090 11230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:46.474618 11230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:46.474663 11230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:46.474675 11230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:46.474686 11230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:46.474696 11230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:46.476289 11230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:36565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:46.476462 11230 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:46.476583 11230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:46.479423 11238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:46.479391 11236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:46.479404 11235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:46.479440 11230 server_base.cc:1060] running on GCE node
I20260319 16:37:46.479810 11230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:46.480036 11230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:46.481181 11230 hybrid_clock.cc:648] HybridClock initialized: now 1773938266481164 us; error 33 us; skew 500 ppm
Mar 19 16:37:46 dist-test-slave-svn8 krb5kdc[11214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:46.483752 11230 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:37:46.484334 11230 webserver.cc:492] Webserver started at http://127.24.187.254:37363/ using document root <none> and password file <none>
I20260319 16:37:46.484542 11230 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:46.484586 11230 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:46.484694 11230 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:46.485870 11230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "49130078ed594b858b168322cf274800"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "83cc1fb14d30980d4963cc52e9078a6b"
server_key_iv: "b620cbc2f846b331dd8a0e2617d03837"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.486141 11230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "49130078ed594b858b168322cf274800"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "83cc1fb14d30980d4963cc52e9078a6b"
server_key_iv: "b620cbc2f846b331dd8a0e2617d03837"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.487181 11230 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:46.487891 11245 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:46.488099 11230 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:46.488157 11230 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "49130078ed594b858b168322cf274800"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "83cc1fb14d30980d4963cc52e9078a6b"
server_key_iv: "b620cbc2f846b331dd8a0e2617d03837"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.488209 11230 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:46.502523 11230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:46.504280 11230 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:46.504383 11230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:46.508294 11230 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:36565
I20260319 16:37:46.508337 11297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:36565 every 8 connection(s)
I20260319 16:37:46.508608 11230 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:46.509152 11298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:46.511488 11298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800: Bootstrap starting.
I20260319 16:37:46.512050 11298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:46.512225 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11230
I20260319 16:37:46.512271 11298 log.cc:826] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:46.512342 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:46.512521 25327 external_mini_cluster.cc:1468] Setting key a9e6359b671ab2276349e678c32da041
I20260319 16:37:46.512938 11298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800: No bootstrap required, opened a new log
I20260319 16:37:46.514272 11298 raft_consensus.cc:359] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49130078ed594b858b168322cf274800" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36565 } }
I20260319 16:37:46.514400 11298 raft_consensus.cc:385] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:46.514429 11298 raft_consensus.cc:740] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49130078ed594b858b168322cf274800, State: Initialized, Role: FOLLOWER
I20260319 16:37:46.514549 11298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49130078ed594b858b168322cf274800" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36565 } }
I20260319 16:37:46.514621 11298 raft_consensus.cc:399] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:46.514649 11298 raft_consensus.cc:493] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:46.514695 11298 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:46.515198 11298 raft_consensus.cc:515] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49130078ed594b858b168322cf274800" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36565 } }
I20260319 16:37:46.515296 11298 leader_election.cc:304] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49130078ed594b858b168322cf274800; no voters:
I20260319 16:37:46.515432 11298 leader_election.cc:290] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:46.515487 11303 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:46.515638 11303 raft_consensus.cc:697] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [term 1 LEADER]: Becoming Leader. State: Replica: 49130078ed594b858b168322cf274800, State: Running, Role: LEADER
I20260319 16:37:46.515679 11298 sys_catalog.cc:565] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:46.515779 11303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49130078ed594b858b168322cf274800" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36565 } }
I20260319 16:37:46.516144 11303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49130078ed594b858b168322cf274800. Latest consensus state: current_term: 1 leader_uuid: "49130078ed594b858b168322cf274800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49130078ed594b858b168322cf274800" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36565 } } }
I20260319 16:37:46.516153 11305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49130078ed594b858b168322cf274800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49130078ed594b858b168322cf274800" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36565 } } }
I20260319 16:37:46.516315 11305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:46.516508 11303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:46.516815 11310 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:46.517323 11310 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:46.519083 11310 catalog_manager.cc:1357] Generated new cluster ID: 9ddae8678f234f8f998b9f60bc59b69b
I20260319 16:37:46.519135 11310 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:37:46 dist-test-slave-svn8 krb5kdc[11214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:46.522360 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:46.513046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.24.187.254:36565)
0319 16:37:46.513240 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:46.513249 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:46.513267 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:37:46.514257 (+ 990us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:46.514266 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:46.514358 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:46.514501 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:46.515316 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.516412 (+ 1096us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:46.517617 (+ 1205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.517875 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:46.519620 (+ 1745us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:46.519625 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:46.519626 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:46.519641 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:46.521101 (+ 1460us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:46.521419 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:46.521421 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:46.521422 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:46.521478 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:46.521619 (+ 141us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:46.521620 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:46.521620 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:46.521829 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:46.521948 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:46.522072 (+ 124us) server_negotiation.cc:300] Negotiation successful
0319 16:37:46.522188 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260319 16:37:46.536039 11310 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:46.536787 11310 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:46.548961 11310 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800: Generated new TSK 0
I20260319 16:37:46.549150 11310 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:46.600908 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36565
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:46.675540 11326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:46.675710 11326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:46.675732 11326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:46.677217 11326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:46.677266 11326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:46.677291 11326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:46.679075 11326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:46.679279 11326 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:46.679450 11326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:46.681996 11332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:46.681996 11334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:46.682025 11331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:46.682600 11326 server_base.cc:1060] running on GCE node
I20260319 16:37:46.682749 11326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:46.682919 11326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:46.684083 11326 hybrid_clock.cc:648] HybridClock initialized: now 1773938266684067 us; error 31 us; skew 500 ppm
Mar 19 16:37:46 dist-test-slave-svn8 krb5kdc[11214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:46.686620 11326 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:37:46.687175 11326 webserver.cc:492] Webserver started at http://127.24.187.193:41727/ using document root <none> and password file <none>
I20260319 16:37:46.687361 11326 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:46.687397 11326 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:46.687497 11326 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:46.688428 11326 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "1eaa3995984e42afa222b6eff869aa91"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "6ce4bb5589df3a006f5e03cca9fc3724"
server_key_iv: "d872c1bb98e7366291f7bee0e49207b5"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.688732 11326 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "1eaa3995984e42afa222b6eff869aa91"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "6ce4bb5589df3a006f5e03cca9fc3724"
server_key_iv: "d872c1bb98e7366291f7bee0e49207b5"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.689962 11326 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:37:46.690740 11341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:46.690936 11326 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:46.691015 11326 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "1eaa3995984e42afa222b6eff869aa91"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "6ce4bb5589df3a006f5e03cca9fc3724"
server_key_iv: "d872c1bb98e7366291f7bee0e49207b5"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.691082 11326 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:46.702911 11326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:46.704704 11326 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:46.704833 11326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:46.705032 11326 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:46.705350 11326 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:46.705384 11326 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:46.705413 11326 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:46.705437 11326 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:46.711105 11326 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:45949
I20260319 16:37:46.711172 11454 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:45949 every 8 connection(s)
I20260319 16:37:46.711488 11326 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:37:46.715451 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11326
I20260319 16:37:46.715565 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:37:46.715762 25327 external_mini_cluster.cc:1468] Setting key 46ce917fa3f5102a457429e683d61d0e
Mar 19 16:37:46 dist-test-slave-svn8 krb5kdc[11214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:46.720500 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:46.712182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:41195 (local address 127.24.187.254:36565)
0319 16:37:46.712234 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:46.712236 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:46.712382 (+ 146us) server_negotiation.cc:408] Connection header received
0319 16:37:46.712907 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:46.712909 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:46.712925 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:46.712980 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:46.713826 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.714314 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:46.714849 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.714969 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:46.717827 (+ 2858us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:46.717830 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:46.717831 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:46.717837 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:46.719222 (+ 1385us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:46.719609 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:46.719610 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:46.719610 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:46.719637 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:46.719796 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:46.719797 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:46.719797 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:46.719917 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:46.719984 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:46.720306 (+ 322us) server_negotiation.cc:300] Negotiation successful
0319 16:37:46.720410 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:37:46.720532 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:46.712117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36565 (local address 127.24.187.193:41195)
0319 16:37:46.712272 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:46.712305 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:46.712814 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:46.713043 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:46.713047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:46.713374 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:46.713660 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:46.713665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.714400 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:46.714400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:46.714774 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:46.714775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.714867 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:46.715309 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:46.715333 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:46.717739 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:46.719319 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:46.719344 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:46.719346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:46.719537 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:46.719694 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:46.719694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:46.719694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:46.719743 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:46.720013 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:46.720019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:46.720162 (+ 143us) client_negotiation.cc:769] Sending connection context
0319 16:37:46.720269 (+ 107us) client_negotiation.cc:240] Negotiation successful
0319 16:37:46.720380 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260319 16:37:46.720955 11455 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36565
I20260319 16:37:46.721048 11455 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:46.721225 11455 heartbeater.cc:507] Master 127.24.187.254:36565 requested a full tablet report, sending...
I20260319 16:37:46.721728 11262 ts_manager.cc:194] Registered new tserver with Master: 1eaa3995984e42afa222b6eff869aa91 (127.24.187.193:45949)
I20260319 16:37:46.722357 11262 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:41195
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:46.764338 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36565
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:46.840824 11462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:46.841001 11462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:46.841032 11462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:46.842479 11462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:46.842520 11462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:46.842553 11462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:46.844141 11462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:46.844355 11462 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:46.844519 11462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:46.847011 11470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:46.847084 11467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:46.847100 11468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:46.847309 11462 server_base.cc:1060] running on GCE node
I20260319 16:37:46.847589 11462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:46.847811 11462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:46.848980 11462 hybrid_clock.cc:648] HybridClock initialized: now 1773938266848964 us; error 26 us; skew 500 ppm
Mar 19 16:37:46 dist-test-slave-svn8 krb5kdc[11214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:46.851600 11462 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:37:46.852114 11462 webserver.cc:492] Webserver started at http://127.24.187.194:33503/ using document root <none> and password file <none>
I20260319 16:37:46.852288 11462 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:46.852324 11462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:46.852438 11462 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:46.853379 11462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "e1a452590c40460c81a0bb0844346433"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "7e99b08ca50faefa5cab0beeb06130ff"
server_key_iv: "8f681ec27c2baf7ecc07f38c2383eadb"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.853669 11462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "e1a452590c40460c81a0bb0844346433"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "7e99b08ca50faefa5cab0beeb06130ff"
server_key_iv: "8f681ec27c2baf7ecc07f38c2383eadb"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.854839 11462 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:46.855567 11477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:46.855732 11462 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:46.855791 11462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "e1a452590c40460c81a0bb0844346433"
format_stamp: "Formatted at 2026-03-19 16:37:46 on dist-test-slave-svn8"
server_key: "7e99b08ca50faefa5cab0beeb06130ff"
server_key_iv: "8f681ec27c2baf7ecc07f38c2383eadb"
server_key_version: "encryptionkey@0"
I20260319 16:37:46.855854 11462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:46.864900 11462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:46.866842 11462 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:46.866981 11462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:46.867226 11462 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:46.867547 11462 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:46.867579 11462 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:46.867609 11462 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:46.867625 11462 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:46.872970 11462 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:39023
I20260319 16:37:46.873039 11591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:39023 every 8 connection(s)
I20260319 16:37:46.873304 11462 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:37:46 dist-test-slave-svn8 krb5kdc[11214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:46.879088 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11462
I20260319 16:37:46.879202 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:37:46.879374 25327 external_mini_cluster.cc:1468] Setting key 54b39aa68f2584d0768121c49a4b1ad5
I20260319 16:37:46.882125 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:46.874058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:58931 (local address 127.24.187.254:36565)
0319 16:37:46.874132 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:46.874135 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:46.874223 (+ 88us) server_negotiation.cc:408] Connection header received
0319 16:37:46.874736 (+ 513us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:46.874737 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:46.874747 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:46.874799 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:46.875492 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.876060 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:46.876606 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.876728 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:46.879010 (+ 2282us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:46.879015 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:46.879015 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:46.879024 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:46.880694 (+ 1670us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:46.881117 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:46.881118 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:46.881119 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:46.881145 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:46.881379 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:46.881380 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:46.881380 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:46.881488 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:46.881552 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:46.881934 (+ 382us) server_negotiation.cc:300] Negotiation successful
0319 16:37:46.882032 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260319 16:37:46.882246 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:46.874001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36565 (local address 127.24.187.194:58931)
0319 16:37:46.874126 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:46.874154 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:46.874650 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:46.874841 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:46.874844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:46.875114 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:46.875384 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:46.875386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.876134 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:46.876135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:46.876491 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:46.876493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:46.876578 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:46.877036 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:46.877052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:46.878920 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:46.880799 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:46.880806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:46.880808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:46.881032 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:46.881234 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:46.881235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:46.881236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:46.881308 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:46.881602 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:46.881608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:46.881773 (+ 165us) client_negotiation.cc:769] Sending connection context
0319 16:37:46.881897 (+ 124us) client_negotiation.cc:240] Negotiation successful
0319 16:37:46.882051 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260319 16:37:46.882900 11592 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36565
I20260319 16:37:46.883015 11592 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:46.883227 11592 heartbeater.cc:507] Master 127.24.187.254:36565 requested a full tablet report, sending...
I20260319 16:37:46.883695 11262 ts_manager.cc:194] Registered new tserver with Master: e1a452590c40460c81a0bb0844346433 (127.24.187.194:39023)
I20260319 16:37:46.884083 11262 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:58931
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:46.928579 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:36565
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:47.004784 11599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:47.004999 11599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:47.005023 11599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:47.006650 11599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:47.006706 11599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:47.006735 11599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:47.008455 11599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:47.008664 11599 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:47.008838 11599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:47.011190 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.011201 11604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.011207 11605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:47.011740 11599 server_base.cc:1060] running on GCE node
I20260319 16:37:47.011894 11599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:47.012094 11599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:47.013257 11599 hybrid_clock.cc:648] HybridClock initialized: now 1773938267013232 us; error 29 us; skew 500 ppm
Mar 19 16:37:47 dist-test-slave-svn8 krb5kdc[11214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938267, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:47.015794 11599 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:37:47.016372 11599 webserver.cc:492] Webserver started at http://127.24.187.195:39137/ using document root <none> and password file <none>
I20260319 16:37:47.016566 11599 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:47.016605 11599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:47.016710 11599 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:47.017653 11599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "626884207d9a476d8fccaa539ad2a672"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "d5bf635ad519f1e22f47a777965c7fae"
server_key_iv: "0c03b9aa9ddf792d9ffcef9b304bab55"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.018002 11599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "626884207d9a476d8fccaa539ad2a672"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "d5bf635ad519f1e22f47a777965c7fae"
server_key_iv: "0c03b9aa9ddf792d9ffcef9b304bab55"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.019186 11599 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:47.020015 11614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.020222 11599 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:47.020288 11599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "626884207d9a476d8fccaa539ad2a672"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "d5bf635ad519f1e22f47a777965c7fae"
server_key_iv: "0c03b9aa9ddf792d9ffcef9b304bab55"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.020354 11599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:47.033097 11599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:47.034976 11599 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:47.035097 11599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:47.035295 11599 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:47.035633 11599 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:47.035666 11599 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.035696 11599 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:47.035712 11599 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.041196 11599 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:35511
I20260319 16:37:47.041268 11727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:35511 every 8 connection(s)
I20260319 16:37:47.041525 11599 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:47.043221 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11599
I20260319 16:37:47.043303 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:37:47.043466 25327 external_mini_cluster.cc:1468] Setting key ff954970ff33dbc8056d8d5dbc765584
Mar 19 16:37:47 dist-test-slave-svn8 krb5kdc[11214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938267, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:47.049847 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.042274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:43865 (local address 127.24.187.254:36565)
0319 16:37:47.042388 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.042390 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.042402 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:37:47.042886 (+ 484us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.042888 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.042899 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.042963 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:47.043679 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.044396 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.044904 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.045010 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.047385 (+ 2375us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:47.047388 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:47.047389 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:47.047396 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:47.048579 (+ 1183us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:47.048985 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:47.048987 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:47.048987 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:47.049015 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:47.049200 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:47.049201 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:47.049202 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:47.049309 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:47.049387 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.049643 (+ 256us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.049751 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260319 16:37:47.049947 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.042156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:36565 (local address 127.24.187.195:43865)
0319 16:37:47.042281 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.042311 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.042780 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.043004 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.043007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.043275 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:47.043591 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.043595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.044466 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.044467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.044842 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.044843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.044935 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.045348 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:47.045374 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:47.047285 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:47.048686 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:47.048693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:47.048695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:47.048921 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:47.049074 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:47.049075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:47.049075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:47.049144 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:47.049375 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:47.049381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:47.049515 (+ 134us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.049607 (+ 92us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.049755 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":66,"threads_started":1}
I20260319 16:37:47.050403 11728 heartbeater.cc:344] Connected to a master server at 127.24.187.254:36565
I20260319 16:37:47.050486 11728 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:47.050679 11728 heartbeater.cc:507] Master 127.24.187.254:36565 requested a full tablet report, sending...
I20260319 16:37:47.051069 11262 ts_manager.cc:194] Registered new tserver with Master: 626884207d9a476d8fccaa539ad2a672 (127.24.187.195:35511)
I20260319 16:37:47.051438 11262 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:43865
I20260319 16:37:47.055472 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:37:47.062369 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.056838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.24.187.254:36565)
0319 16:37:47.056891 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.056893 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.056904 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:37:47.057001 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.057002 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.057013 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.057069 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:47.057805 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.058289 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.058943 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.059070 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.060082 (+ 1012us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:47.060085 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:47.060086 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:47.060096 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:47.061400 (+ 1304us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:47.061743 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:47.061744 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:47.061744 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:47.061769 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:47.061936 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:47.061936 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:47.061937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:47.062046 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:47.062118 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.062200 (+ 82us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.062262 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260319 16:37:47.063870 11262 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:37:47.065208 11262 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:37:47.070966 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.067977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:39023 (local address 127.0.0.1:47160)
0319 16:37:47.068328 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.068338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.068399 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.068607 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.068610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.068613 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.068905 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.068907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.069881 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.069882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.070679 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.070681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.070750 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.070800 (+ 50us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.070846 (+ 46us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.070885 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":89,"threads_started":1}
I20260319 16:37:47.071321 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.068101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:45949 (local address 127.0.0.1:40964)
0319 16:37:47.068481 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.068490 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.068553 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.068812 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.068813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.068815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.069020 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.069023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.070344 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.070345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.071109 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.071112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.071178 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.071182 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.071207 (+ 25us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.071242 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":76,"threads_started":1}
I20260319 16:37:47.071972 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.068101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.24.187.194:39023)
0319 16:37:47.068260 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.068271 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.068406 (+ 135us) server_negotiation.cc:408] Connection header received
0319 16:37:47.068468 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.068470 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.068487 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.068896 (+ 409us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.068980 (+ 84us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.069802 (+ 822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.071203 (+ 1401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.071761 (+ 558us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.071820 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.071852 (+ 32us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.071897 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260319 16:37:47.071986 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.068060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.24.187.195:35511)
0319 16:37:47.068218 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.068225 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.068403 (+ 178us) server_negotiation.cc:408] Connection header received
0319 16:37:47.068448 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.068450 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.068466 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.068525 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.068988 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.070140 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.071314 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.071818 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.071852 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.071879 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.071916 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260319 16:37:47.072139 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.067891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:35511 (local address 127.0.0.1:41824)
0319 16:37:47.068223 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.068256 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.068341 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.068643 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.068644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.068646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.068905 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.068907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.070230 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.070231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.071075 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.071078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.071141 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.071144 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.071170 (+ 26us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.071209 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":73,"threads_started":1}
I20260319 16:37:47.072270 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.068326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.24.187.193:45949)
0319 16:37:47.068531 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.068539 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.068548 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:47.068650 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.068652 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.068667 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.068724 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.069113 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.070264 (+ 1151us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.071453 (+ 1189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.071925 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.071961 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.071987 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.072018 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":117,"threads_started":1}
I20260319 16:37:47.072530 11525 tablet_service.cc:1511] Processing CreateTablet for tablet e122781f5e1e4109823b3f6eea078958 (DEFAULT_TABLE table=test-table [id=8b33f2a2b16b443c8cdcd8a4dfa68020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:47.072794 11525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e122781f5e1e4109823b3f6eea078958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:47.072757 11662 tablet_service.cc:1511] Processing CreateTablet for tablet e122781f5e1e4109823b3f6eea078958 (DEFAULT_TABLE table=test-table [id=8b33f2a2b16b443c8cdcd8a4dfa68020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:47.072989 11662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e122781f5e1e4109823b3f6eea078958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:47.075023 11745 tablet_bootstrap.cc:492] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Bootstrap starting.
I20260319 16:37:47.075343 11389 tablet_service.cc:1511] Processing CreateTablet for tablet e122781f5e1e4109823b3f6eea078958 (DEFAULT_TABLE table=test-table [id=8b33f2a2b16b443c8cdcd8a4dfa68020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:47.075605 11389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e122781f5e1e4109823b3f6eea078958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:47.075685 11745 tablet_bootstrap.cc:654] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:47.075968 11745 log.cc:826] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:47.076749 11745 tablet_bootstrap.cc:492] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: No bootstrap required, opened a new log
I20260319 16:37:47.076817 11745 ts_tablet_manager.cc:1403] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:47.078217 11748 tablet_bootstrap.cc:492] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: Bootstrap starting.
I20260319 16:37:47.078328 11746 tablet_bootstrap.cc:492] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Bootstrap starting.
I20260319 16:37:47.078359 11745 raft_consensus.cc:359] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.078493 11745 raft_consensus.cc:385] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:47.078521 11745 raft_consensus.cc:740] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a452590c40460c81a0bb0844346433, State: Initialized, Role: FOLLOWER
I20260319 16:37:47.078788 11748 tablet_bootstrap.cc:654] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:47.078646 11745 consensus_queue.cc:260] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.078912 11746 tablet_bootstrap.cc:654] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:47.079082 11748 log.cc:826] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:47.079137 11745 ts_tablet_manager.cc:1434] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260319 16:37:47.079193 11746 log.cc:826] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:47.079254 11592 heartbeater.cc:499] Master 127.24.187.254:36565 was elected leader, sending a full tablet report...
I20260319 16:37:47.079967 11748 tablet_bootstrap.cc:492] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: No bootstrap required, opened a new log
I20260319 16:37:47.080065 11748 ts_tablet_manager.cc:1403] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:47.080142 11746 tablet_bootstrap.cc:492] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: No bootstrap required, opened a new log
I20260319 16:37:47.080209 11746 ts_tablet_manager.cc:1403] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:47.081310 11748 raft_consensus.cc:359] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.081427 11748 raft_consensus.cc:385] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:47.081447 11748 raft_consensus.cc:740] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaa3995984e42afa222b6eff869aa91, State: Initialized, Role: FOLLOWER
I20260319 16:37:47.081370 11746 raft_consensus.cc:359] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.081482 11746 raft_consensus.cc:385] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:47.081521 11746 raft_consensus.cc:740] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626884207d9a476d8fccaa539ad2a672, State: Initialized, Role: FOLLOWER
I20260319 16:37:47.081619 11746 consensus_queue.cc:260] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.081547 11748 consensus_queue.cc:260] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.081888 11746 ts_tablet_manager.cc:1434] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:37:47.081941 11728 heartbeater.cc:499] Master 127.24.187.254:36565 was elected leader, sending a full tablet report...
I20260319 16:37:47.082011 11748 ts_tablet_manager.cc:1434] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:37:47.082125 11455 heartbeater.cc:499] Master 127.24.187.254:36565 was elected leader, sending a full tablet report...
W20260319 16:37:47.123842 11593 tablet.cc:2379] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:47.201509 11753 raft_consensus.cc:493] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:47.201658 11753 raft_consensus.cc:515] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.202054 11753 leader_election.cc:290] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 626884207d9a476d8fccaa539ad2a672 (127.24.187.195:35511), e1a452590c40460c81a0bb0844346433 (127.24.187.194:39023)
I20260319 16:37:47.205273 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.202256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:35511 (local address 127.24.187.193:35225)
0319 16:37:47.202327 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.202339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.202384 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.202557 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.202558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.202561 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.202843 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.202846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.204121 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.204122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.205025 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.205027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.205106 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.205111 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.205143 (+ 32us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.205181 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260319 16:37:47.205287 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.202272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:39023 (local address 127.24.187.193:40561)
0319 16:37:47.202458 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.202467 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.202519 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.202672 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.202672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.202674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.202942 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.202943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.204075 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.204076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.205045 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.205047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.205121 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.205124 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.205155 (+ 31us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.205181 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":117,"threads_started":1}
I20260319 16:37:47.205739 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.202329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:35225 (local address 127.24.187.195:35511)
0319 16:37:47.202388 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.202390 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.202439 (+ 49us) server_negotiation.cc:408] Connection header received
0319 16:37:47.202470 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.202471 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.202481 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.202531 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.202964 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.204037 (+ 1073us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.205120 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.205586 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.205596 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.205618 (+ 22us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.205649 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260319 16:37:47.205775 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.202365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:40561 (local address 127.24.187.194:39023)
0319 16:37:47.202413 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.202416 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.202528 (+ 112us) server_negotiation.cc:408] Connection header received
0319 16:37:47.202574 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.202575 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.202603 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.202645 (+ 42us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.203147 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.203964 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.205120 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.205610 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.205616 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.205635 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.205663 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:37:47.206017 11682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e122781f5e1e4109823b3f6eea078958" candidate_uuid: "1eaa3995984e42afa222b6eff869aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626884207d9a476d8fccaa539ad2a672" is_pre_election: true
I20260319 16:37:47.206038 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e122781f5e1e4109823b3f6eea078958" candidate_uuid: "1eaa3995984e42afa222b6eff869aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a452590c40460c81a0bb0844346433" is_pre_election: true
I20260319 16:37:47.206138 11682 raft_consensus.cc:2468] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaa3995984e42afa222b6eff869aa91 in term 0.
I20260319 16:37:47.206176 11546 raft_consensus.cc:2468] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eaa3995984e42afa222b6eff869aa91 in term 0.
I20260319 16:37:47.206399 11345 leader_election.cc:304] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaa3995984e42afa222b6eff869aa91, 626884207d9a476d8fccaa539ad2a672; no voters:
I20260319 16:37:47.206528 11753 raft_consensus.cc:2804] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:47.206598 11753 raft_consensus.cc:493] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:47.206616 11753 raft_consensus.cc:3060] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:47.207293 11753 raft_consensus.cc:515] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.207432 11753 leader_election.cc:290] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [CANDIDATE]: Term 1 election: Requested vote from peers 626884207d9a476d8fccaa539ad2a672 (127.24.187.195:35511), e1a452590c40460c81a0bb0844346433 (127.24.187.194:39023)
I20260319 16:37:47.207610 11682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e122781f5e1e4109823b3f6eea078958" candidate_uuid: "1eaa3995984e42afa222b6eff869aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626884207d9a476d8fccaa539ad2a672"
I20260319 16:37:47.207659 11546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e122781f5e1e4109823b3f6eea078958" candidate_uuid: "1eaa3995984e42afa222b6eff869aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a452590c40460c81a0bb0844346433"
I20260319 16:37:47.207679 11682 raft_consensus.cc:3060] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:47.207748 11546 raft_consensus.cc:3060] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:47.208894 11682 raft_consensus.cc:2468] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaa3995984e42afa222b6eff869aa91 in term 1.
I20260319 16:37:47.208933 11546 raft_consensus.cc:2468] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eaa3995984e42afa222b6eff869aa91 in term 1.
I20260319 16:37:47.209100 11345 leader_election.cc:304] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaa3995984e42afa222b6eff869aa91, 626884207d9a476d8fccaa539ad2a672; no voters:
I20260319 16:37:47.209201 11753 raft_consensus.cc:2804] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:47.209362 11753 raft_consensus.cc:697] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 1 LEADER]: Becoming Leader. State: Replica: 1eaa3995984e42afa222b6eff869aa91, State: Running, Role: LEADER
I20260319 16:37:47.209460 11753 consensus_queue.cc:237] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } }
I20260319 16:37:47.210289 11261 catalog_manager.cc:5671] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eaa3995984e42afa222b6eff869aa91 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "1eaa3995984e42afa222b6eff869aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaa3995984e42afa222b6eff869aa91" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 45949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 } health_report { overall_health: UNKNOWN } } }
W20260319 16:37:47.212131 11456 tablet.cc:2379] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:47.217423 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.215355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.24.187.193:45949)
0319 16:37:47.215419 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.215420 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.215427 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:37:47.215460 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.215460 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.215476 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.215521 (+ 45us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:47.215780 (+ 259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.216261 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.216852 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.217004 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.217060 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:47.217166 (+ 106us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:37:47.217236 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.217274 (+ 38us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.217346 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260319 16:37:47.219081 11682 raft_consensus.cc:1275] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 1 FOLLOWER]: Refusing update from remote peer 1eaa3995984e42afa222b6eff869aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:47.219082 11546 raft_consensus.cc:1275] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 1 FOLLOWER]: Refusing update from remote peer 1eaa3995984e42afa222b6eff869aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:47.219386 11755 consensus_queue.cc:1048] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a452590c40460c81a0bb0844346433" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 39023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:47.219527 11753 consensus_queue.cc:1048] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "626884207d9a476d8fccaa539ad2a672" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:47.221482 11758 mvcc.cc:204] Tried to move back new op lower bound from 7266051142527049728 to 7266051142490714112. Current Snapshot: MvccSnapshot[applied={T|T < 7266051142527049728}]
I20260319 16:37:47.221957 11759 mvcc.cc:204] Tried to move back new op lower bound from 7266051142527049728 to 7266051142490714112. Current Snapshot: MvccSnapshot[applied={T|T < 7266051142527049728}]
I20260319 16:37:47.222657 11760 mvcc.cc:204] Tried to move back new op lower bound from 7266051142527049728 to 7266051142490714112. Current Snapshot: MvccSnapshot[applied={T|T < 7266051142527049728 or (T in {7266051142527049728})}]
I20260319 16:37:47.223917 11262 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:47.224025 11262 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:47.224682 11262 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 49130078ed594b858b168322cf274800: Sending DeleteTablet for 3 replicas of tablet e122781f5e1e4109823b3f6eea078958
I20260319 16:37:47.224987 11525 tablet_service.cc:1558] Processing DeleteTablet for tablet e122781f5e1e4109823b3f6eea078958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:47 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:47160
I20260319 16:37:47.224996 11389 tablet_service.cc:1558] Processing DeleteTablet for tablet e122781f5e1e4109823b3f6eea078958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:47 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:40964
I20260319 16:37:47.225205 11768 tablet_replica.cc:333] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91: stopping tablet replica
I20260319 16:37:47.225281 11768 raft_consensus.cc:2243] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:37:47.225366 11768 raft_consensus.cc:2272] T e122781f5e1e4109823b3f6eea078958 P 1eaa3995984e42afa222b6eff869aa91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:47.225349 11662 tablet_service.cc:1558] Processing DeleteTablet for tablet e122781f5e1e4109823b3f6eea078958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:47 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:41824
I20260319 16:37:47.225525 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11326
I20260319 16:37:47.225590 11770 tablet_replica.cc:333] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: stopping tablet replica
I20260319 16:37:47.226361 11770 raft_consensus.cc:2243] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:47.226469 11770 raft_consensus.cc:2272] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:47.226812 11770 ts_tablet_manager.cc:1916] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:47.227425 11769 tablet_replica.cc:333] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: stopping tablet replica
I20260319 16:37:47.227514 11769 raft_consensus.cc:2243] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:47.227614 11769 raft_consensus.cc:2272] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:47.228040 11769 ts_tablet_manager.cc:1916] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:47.228209 11770 ts_tablet_manager.cc:1929] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:47.228271 11770 log.cc:1199] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/e122781f5e1e4109823b3f6eea078958
I20260319 16:37:47.228536 11770 ts_tablet_manager.cc:1950] T e122781f5e1e4109823b3f6eea078958 P 626884207d9a476d8fccaa539ad2a672: Deleting consensus metadata
I20260319 16:37:47.228916 11249 catalog_manager.cc:5002] TS 626884207d9a476d8fccaa539ad2a672 (127.24.187.195:35511): tablet e122781f5e1e4109823b3f6eea078958 (table test-table [id=8b33f2a2b16b443c8cdcd8a4dfa68020]) successfully deleted
I20260319 16:37:47.229369 11769 ts_tablet_manager.cc:1929] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:47.229421 11769 log.cc:1199] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/e122781f5e1e4109823b3f6eea078958
I20260319 16:37:47.229638 11769 ts_tablet_manager.cc:1950] T e122781f5e1e4109823b3f6eea078958 P e1a452590c40460c81a0bb0844346433: Deleting consensus metadata
I20260319 16:37:47.229971 11247 catalog_manager.cc:5002] TS e1a452590c40460c81a0bb0844346433 (127.24.187.194:39023): tablet e122781f5e1e4109823b3f6eea078958 (table test-table [id=8b33f2a2b16b443c8cdcd8a4dfa68020]) successfully deleted
W20260319 16:37:47.231381 11249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:45949 (error 108)
I20260319 16:37:47.231559 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11462
I20260319 16:37:47.231806 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.231709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:45949 (local address 127.0.0.1:40974)
0319 16:37:47.231754 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.231777 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:45949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260319 16:37:47.231882 11249 catalog_manager.cc:4729] TS 1eaa3995984e42afa222b6eff869aa91 (127.24.187.193:45949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e122781f5e1e4109823b3f6eea078958: Network error: Client connection negotiation failed: client connection to 127.24.187.193:45949: connect: Connection refused (error 111)
I20260319 16:37:47.236950 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11599
I20260319 16:37:47.240993 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11230
2026-03-19T16:37:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2833 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-03-19T16:37:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:47Z Disabled control of system clock
I20260319 16:37:47.267396 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38903
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:38903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:47.341847 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:47.342018 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:47.342037 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:47.343417 11775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:47.343456 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:47.343467 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:47.343477 11775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:47.343487 11775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:47.344941 11775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:38903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:38903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:47.345111 11775 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:47.345278 11775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:47.347425 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.347470 11781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.347522 11783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:47.347740 11775 server_base.cc:1060] running on GCE node
I20260319 16:37:47.347913 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:47.348130 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:47.349256 11775 hybrid_clock.cc:648] HybridClock initialized: now 1773938267349237 us; error 34 us; skew 500 ppm
I20260319 16:37:47.350301 11775 webserver.cc:492] Webserver started at http://127.24.187.254:41799/ using document root <none> and password file <none>
I20260319 16:37:47.350509 11775 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:47.350549 11775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:47.350685 11775 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:47.351541 11775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "2d6ccfadcc64487daa95a79c5a213ef9"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "f4620c82ec07321492e00004528cf5ba"
server_key_iv: "b87cb293094685f896d6908e6cf33149"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.351822 11775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "2d6ccfadcc64487daa95a79c5a213ef9"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "f4620c82ec07321492e00004528cf5ba"
server_key_iv: "b87cb293094685f896d6908e6cf33149"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.352907 11775 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:47.353739 11789 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.353909 11775 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:47.353973 11775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2d6ccfadcc64487daa95a79c5a213ef9"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "f4620c82ec07321492e00004528cf5ba"
server_key_iv: "b87cb293094685f896d6908e6cf33149"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.354040 11775 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:47.367852 11775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:47.368098 11775 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:47.368216 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:47.372193 11775 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:38903
I20260319 16:37:47.372222 11841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:38903 every 8 connection(s)
I20260319 16:37:47.372524 11775 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:47.373037 11842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:47.375435 11842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9: Bootstrap starting.
I20260319 16:37:47.375946 11842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:47.376140 11842 log.cc:826] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:47.376713 11842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9: No bootstrap required, opened a new log
I20260319 16:37:47.377918 11842 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38903 } }
I20260319 16:37:47.378034 11842 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:47.378053 11842 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d6ccfadcc64487daa95a79c5a213ef9, State: Initialized, Role: FOLLOWER
I20260319 16:37:47.378151 11842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38903 } }
I20260319 16:37:47.378218 11842 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:47.378240 11842 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:47.378268 11842 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:47.378739 11842 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38903 } }
I20260319 16:37:47.378822 11842 leader_election.cc:304] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d6ccfadcc64487daa95a79c5a213ef9; no voters:
I20260319 16:37:47.378927 11842 leader_election.cc:290] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:47.378993 11845 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:47.379115 11842 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:47.379127 11845 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [term 1 LEADER]: Becoming Leader. State: Replica: 2d6ccfadcc64487daa95a79c5a213ef9, State: Running, Role: LEADER
I20260319 16:37:47.379230 11845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38903 } }
I20260319 16:37:47.379638 11847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d6ccfadcc64487daa95a79c5a213ef9. Latest consensus state: current_term: 1 leader_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38903 } } }
I20260319 16:37:47.379868 11847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:47.379642 11846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d6ccfadcc64487daa95a79c5a213ef9" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 38903 } } }
I20260319 16:37:47.380087 11846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:47.380234 11854 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:47.380661 11854 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:47.381256 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11775
I20260319 16:37:47.381356 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:47.381528 25327 external_mini_cluster.cc:1468] Setting key de4826a8c62d183eb8ca2a2e78a6df90
I20260319 16:37:47.382298 11854 catalog_manager.cc:1357] Generated new cluster ID: 8956c0f66ae74e9797dddb14e58d6821
I20260319 16:37:47.382341 11854 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:37:47.394259 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.381980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.24.187.254:38903)
0319 16:37:47.389915 (+ 7935us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.389922 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.389951 (+ 29us) server_negotiation.cc:408] Connection header received
0319 16:37:47.390070 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.390077 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.390140 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.390274 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:47.390788 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.391963 (+ 1175us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.392840 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.393396 (+ 556us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.393513 (+ 117us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:47.393518 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:47.393518 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:47.393771 (+ 253us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:47.393799 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:47.393806 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:47.393866 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.393956 (+ 90us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.394102 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7823,"thread_start_us":69,"threads_started":1}
I20260319 16:37:47.406840 11854 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:47.407354 11854 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:47.410240 11854 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9: Generated new TSK 0
I20260319 16:37:47.410403 11854 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:37:47.415531 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38903
--builtin_ntp_servers=127.24.187.212:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:47.492595 11866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:47.492765 11866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:47.492786 11866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:47.494331 11866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:47.494377 11866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:47.494401 11866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:47.495957 11866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:47.496148 11866 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:47.496378 11866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:47.498826 11874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.498822 11872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:47.498947 11866 server_base.cc:1060] running on GCE node
W20260319 16:37:47.499202 11871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:47.499402 11866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:47.499643 11866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:47.500810 11866 hybrid_clock.cc:648] HybridClock initialized: now 1773938267500788 us; error 32 us; skew 500 ppm
I20260319 16:37:47.502115 11866 webserver.cc:492] Webserver started at http://127.24.187.193:34779/ using document root <none> and password file <none>
I20260319 16:37:47.502317 11866 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:47.502370 11866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:47.502481 11866 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:47.503686 11866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "cda32e496dc84ff4b988b01d914002dc"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "9100c0b9963c87a4155e658ad6951d36"
server_key_iv: "971c646a91f0470b20065b011d1d3ccf"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.504042 11866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "cda32e496dc84ff4b988b01d914002dc"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "9100c0b9963c87a4155e658ad6951d36"
server_key_iv: "971c646a91f0470b20065b011d1d3ccf"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.505501 11866 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260319 16:37:47.506420 11880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.506603 11866 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:47.506655 11866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "cda32e496dc84ff4b988b01d914002dc"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "9100c0b9963c87a4155e658ad6951d36"
server_key_iv: "971c646a91f0470b20065b011d1d3ccf"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.506719 11866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:47.518086 11866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:47.518319 11866 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:47.518424 11866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:47.518618 11866 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:47.518921 11866 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:47.518954 11866 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.518982 11866 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:47.519009 11866 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.524472 11866 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:42797
I20260319 16:37:47.524564 11993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:42797 every 8 connection(s)
I20260319 16:37:47.524814 11866 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:37:47.529292 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.525556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:48261 (local address 127.24.187.254:38903)
0319 16:37:47.525606 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.525608 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.525817 (+ 209us) server_negotiation.cc:408] Connection header received
0319 16:37:47.526444 (+ 627us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.526445 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.526458 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.526509 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:47.527077 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.527569 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.528341 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.528480 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.529042 (+ 562us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:47.529046 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:47.529046 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:47.529050 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:47.529065 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:47.529068 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:47.529119 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.529196 (+ 77us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.529226 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:37:47.529420 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.525489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38903 (local address 127.24.187.193:48261)
0319 16:37:47.525676 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.525706 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.526321 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.526570 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.526574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.526584 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:47.527001 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.527005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.527661 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.527662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.528234 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.528238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.528364 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.528892 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:47.528893 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:47.528916 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:47.528935 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:47.529130 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:47.529133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:47.529136 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.529194 (+ 58us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.529267 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260319 16:37:47.529903 11994 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38903
I20260319 16:37:47.529989 11994 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:47.530179 11994 heartbeater.cc:507] Master 127.24.187.254:38903 requested a full tablet report, sending...
I20260319 16:37:47.530184 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11866
I20260319 16:37:47.530258 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:37:47.530488 25327 external_mini_cluster.cc:1468] Setting key bb2aea93bc16ad8e3f744fa0fcbf371c
I20260319 16:37:47.530695 11806 ts_manager.cc:194] Registered new tserver with Master: cda32e496dc84ff4b988b01d914002dc (127.24.187.193:42797)
I20260319 16:37:47.531347 11806 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.193:48261
I20260319 16:37:47.531688 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38903
--builtin_ntp_servers=127.24.187.212:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:47.607158 11997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:47.607306 11997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:47.607326 11997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:47.608701 11997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:47.608737 11997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:47.608758 11997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:47.610302 11997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:47.610469 11997 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:47.610639 11997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:47.613041 12005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.613070 12002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.613189 12003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:47.613206 11997 server_base.cc:1060] running on GCE node
I20260319 16:37:47.613497 11997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:47.613667 11997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:47.614794 11997 hybrid_clock.cc:648] HybridClock initialized: now 1773938267614743 us; error 69 us; skew 500 ppm
I20260319 16:37:47.615825 11997 webserver.cc:492] Webserver started at http://127.24.187.194:40663/ using document root <none> and password file <none>
I20260319 16:37:47.615996 11997 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:47.616035 11997 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:47.616114 11997 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:47.617030 11997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "748516283d5740ecb52d9433a34bbe70"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "445de75f55de186ba224c790adaf665f"
server_key_iv: "9905a8bb0ad7a72b68b166c4b4e975ff"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.617285 11997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "748516283d5740ecb52d9433a34bbe70"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "445de75f55de186ba224c790adaf665f"
server_key_iv: "9905a8bb0ad7a72b68b166c4b4e975ff"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.618435 11997 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:37:47.619117 12011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.619266 11997 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:37:47.619323 11997 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "748516283d5740ecb52d9433a34bbe70"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "445de75f55de186ba224c790adaf665f"
server_key_iv: "9905a8bb0ad7a72b68b166c4b4e975ff"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.619395 11997 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:47.657949 11997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:47.658197 11997 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:47.658304 11997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:47.658509 11997 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:47.658823 11997 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:47.658855 11997 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.658883 11997 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:47.658910 11997 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.664674 11997 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:45659
I20260319 16:37:47.664734 12124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:45659 every 8 connection(s)
I20260319 16:37:47.665015 11997 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:37:47.666908 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 11997
I20260319 16:37:47.667002 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:37:47.667210 25327 external_mini_cluster.cc:1468] Setting key 6e77cd757ff43241880eedba87854c75
I20260319 16:37:47.668262 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38903
--builtin_ntp_servers=127.24.187.212:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260319 16:37:47.669551 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.665762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:47589 (local address 127.24.187.254:38903)
0319 16:37:47.665830 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.665833 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.666091 (+ 258us) server_negotiation.cc:408] Connection header received
0319 16:37:47.666606 (+ 515us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.666607 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.666621 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.666679 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:47.667140 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.667778 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.668543 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.668702 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.669299 (+ 597us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:47.669302 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:47.669302 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:47.669307 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:47.669322 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:47.669326 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:47.669391 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.669460 (+ 69us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.669491 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260319 16:37:47.669734 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.665752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38903 (local address 127.24.187.194:47589)
0319 16:37:47.665958 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.665985 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.666492 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.666719 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.666723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.666731 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:47.667035 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.667038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.667925 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.667927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.668464 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.668468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.668598 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.669160 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:47.669161 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:47.669187 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:47.669209 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:47.669395 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:47.669398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:47.669401 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.669460 (+ 59us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.669542 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260319 16:37:47.670212 12125 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38903
I20260319 16:37:47.670316 12125 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:47.670531 12125 heartbeater.cc:507] Master 127.24.187.254:38903 requested a full tablet report, sending...
I20260319 16:37:47.670912 11806 ts_manager.cc:194] Registered new tserver with Master: 748516283d5740ecb52d9433a34bbe70 (127.24.187.194:45659)
I20260319 16:37:47.671277 11806 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.194:47589
W20260319 16:37:47.743530 12128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:47.743688 12128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:47.743716 12128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:47.745151 12128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:47.745199 12128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:47.745232 12128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:47.746904 12128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:47.747116 12128 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:47.747324 12128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:47.749794 12136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:47.749805 12134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:47.750049 12128 server_base.cc:1060] running on GCE node
W20260319 16:37:47.749794 12133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:47.750290 12128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:47.750478 12128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:47.751602 12128 hybrid_clock.cc:648] HybridClock initialized: now 1773938267751589 us; error 27 us; skew 500 ppm
I20260319 16:37:47.752637 12128 webserver.cc:492] Webserver started at http://127.24.187.195:42435/ using document root <none> and password file <none>
I20260319 16:37:47.752790 12128 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:47.752822 12128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:47.752904 12128 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:47.753878 12128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "0809967cb7364f3e92ddd67c63a02e00"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "cdb62fecd6fb2e17d202f766a484ccc1"
server_key_iv: "d41b752e33244b2f711b3dbd3abff97f"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.754141 12128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "0809967cb7364f3e92ddd67c63a02e00"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "cdb62fecd6fb2e17d202f766a484ccc1"
server_key_iv: "d41b752e33244b2f711b3dbd3abff97f"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.755188 12128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:47.755964 12142 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.756138 12128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:37:47.756196 12128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "0809967cb7364f3e92ddd67c63a02e00"
format_stamp: "Formatted at 2026-03-19 16:37:47 on dist-test-slave-svn8"
server_key: "cdb62fecd6fb2e17d202f766a484ccc1"
server_key_iv: "d41b752e33244b2f711b3dbd3abff97f"
server_key_version: "encryptionkey@0"
I20260319 16:37:47.756258 12128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:47.778390 12128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:47.778641 12128 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:47.778752 12128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:47.778946 12128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:47.779276 12128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:47.779315 12128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.779345 12128 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:47.779367 12128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:47.784978 12128 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:37631
I20260319 16:37:47.785040 12255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:37631 every 8 connection(s)
I20260319 16:37:47.785307 12128 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:47.789194 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.786023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:35559 (local address 127.24.187.254:38903)
0319 16:37:47.786079 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.786081 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.786249 (+ 168us) server_negotiation.cc:408] Connection header received
0319 16:37:47.786721 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.786722 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.786731 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.786785 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:47.787237 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.787741 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.788267 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.788395 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.788934 (+ 539us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:47.788937 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:47.788937 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:47.788941 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:47.788955 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:47.788958 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:47.789001 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.789110 (+ 109us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.789137 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:37:47.789300 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.785990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:38903 (local address 127.24.187.195:35559)
0319 16:37:47.786152 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.786177 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.786643 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.786848 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.786851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.786858 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:47.787155 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.787157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.787823 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.787824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.788204 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.788206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.788296 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.788818 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:47.788819 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:47.788837 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:47.788854 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:47.789034 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:47.789039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:47.789042 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.789111 (+ 69us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.789174 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260319 16:37:47.789812 12256 heartbeater.cc:344] Connected to a master server at 127.24.187.254:38903
I20260319 16:37:47.789892 12256 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:47.790069 12256 heartbeater.cc:507] Master 127.24.187.254:38903 requested a full tablet report, sending...
I20260319 16:37:47.790495 11806 ts_manager.cc:194] Registered new tserver with Master: 0809967cb7364f3e92ddd67c63a02e00 (127.24.187.195:37631)
I20260319 16:37:47.790889 11806 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.195:35559
I20260319 16:37:47.793624 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 12128
I20260319 16:37:47.793748 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:37:47.793946 25327 external_mini_cluster.cc:1468] Setting key e79c05c6fcd1043df828dd4c8eaee6eb
I20260319 16:37:47.794947 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:37:47.798789 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.796289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.24.187.254:38903)
0319 16:37:47.796378 (+ 89us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.796380 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.796474 (+ 94us) server_negotiation.cc:408] Connection header received
0319 16:37:47.796532 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.796534 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.796544 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.796588 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:47.796888 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.797367 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.797935 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.798092 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.798528 (+ 436us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:47.798531 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:47.798531 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:47.798535 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:47.798549 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:47.798552 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:47.798619 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.798676 (+ 57us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.798706 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260319 16:37:47.800539 11806 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:37:47.801848 11806 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:37:47.807050 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.803929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:45659 (local address 127.0.0.1:56762)
0319 16:37:47.804485 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.804494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.804544 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.804801 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.804802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.804803 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.804961 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.804962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.806156 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.806157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.806819 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.806821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.806879 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.806916 (+ 37us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.806953 (+ 37us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.806993 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":220,"thread_start_us":66,"threads_started":1}
I20260319 16:37:47.807778 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.803925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:37631 (local address 127.0.0.1:58914)
0319 16:37:47.804291 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.804321 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.804407 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.804659 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.804661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.804664 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.804906 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.804909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.806637 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.806638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.807540 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.807543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.807623 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.807627 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.807660 (+ 33us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.807692 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":90,"threads_started":1}
I20260319 16:37:47.807897 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.803926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42797 (local address 127.0.0.1:38534)
0319 16:37:47.804290 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.804321 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.804406 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.804681 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.804682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.804685 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.804917 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.804919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.806707 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.806708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.807569 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.807571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.807629 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.807631 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.807660 (+ 29us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.807692 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260319 16:37:47.807984 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.804008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.24.187.194:45659)
0319 16:37:47.804388 (+ 380us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.804402 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.804588 (+ 186us) server_negotiation.cc:408] Connection header received
0319 16:37:47.804627 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.804628 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.804666 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.804721 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.805028 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.806081 (+ 1053us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.807241 (+ 1160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.807658 (+ 417us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.807742 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.807770 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.807805 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":108,"threads_started":1}
I20260319 16:37:47.808471 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.804196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.24.187.193:42797)
0319 16:37:47.804472 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.804478 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.804487 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:47.804516 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.804518 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.804531 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.804593 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.804983 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.806574 (+ 1591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.807752 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.808257 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.808331 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.808370 (+ 39us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.808404 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
I20260319 16:37:47.808519 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.804118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.24.187.195:37631)
0319 16:37:47.804299 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.804306 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.804388 (+ 82us) server_negotiation.cc:408] Connection header received
0319 16:37:47.804476 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.804478 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.804495 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.804563 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.804988 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.806532 (+ 1544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.807634 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.808370 (+ 736us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.808421 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.808445 (+ 24us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.808467 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260319 16:37:47.808560 12059 tablet_service.cc:1511] Processing CreateTablet for tablet 1bafd105e5354a02bc89740f6334cfae (DEFAULT_TABLE table=test-table [id=0577f4ac091e43a6b262c943a89e95f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:47.808871 12059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bafd105e5354a02bc89740f6334cfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:47.809085 12190 tablet_service.cc:1511] Processing CreateTablet for tablet 1bafd105e5354a02bc89740f6334cfae (DEFAULT_TABLE table=test-table [id=0577f4ac091e43a6b262c943a89e95f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:47.809101 11928 tablet_service.cc:1511] Processing CreateTablet for tablet 1bafd105e5354a02bc89740f6334cfae (DEFAULT_TABLE table=test-table [id=0577f4ac091e43a6b262c943a89e95f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:47.809341 11928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bafd105e5354a02bc89740f6334cfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:47.809338 12190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bafd105e5354a02bc89740f6334cfae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:47.811663 12270 tablet_bootstrap.cc:492] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc: Bootstrap starting.
I20260319 16:37:47.812032 12272 tablet_bootstrap.cc:492] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Bootstrap starting.
I20260319 16:37:47.812036 12271 tablet_bootstrap.cc:492] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Bootstrap starting.
I20260319 16:37:47.812423 12270 tablet_bootstrap.cc:654] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:47.812587 12272 tablet_bootstrap.cc:654] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:47.812587 12271 tablet_bootstrap.cc:654] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:47.812686 12270 log.cc:826] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:47.812879 12271 log.cc:826] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:47.812879 12272 log.cc:826] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:47.813308 12270 tablet_bootstrap.cc:492] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc: No bootstrap required, opened a new log
I20260319 16:37:47.813364 12270 ts_tablet_manager.cc:1403] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:47.814242 12272 tablet_bootstrap.cc:492] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: No bootstrap required, opened a new log
I20260319 16:37:47.814285 12271 tablet_bootstrap.cc:492] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: No bootstrap required, opened a new log
I20260319 16:37:47.814311 12272 ts_tablet_manager.cc:1403] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:47.814338 12271 ts_tablet_manager.cc:1403] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:47.814734 12270 raft_consensus.cc:359] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.814875 12270 raft_consensus.cc:385] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:47.814901 12270 raft_consensus.cc:740] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda32e496dc84ff4b988b01d914002dc, State: Initialized, Role: FOLLOWER
I20260319 16:37:47.815011 12270 consensus_queue.cc:260] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.815246 12270 ts_tablet_manager.cc:1434] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:47.815299 11994 heartbeater.cc:499] Master 127.24.187.254:38903 was elected leader, sending a full tablet report...
I20260319 16:37:47.815644 12271 raft_consensus.cc:359] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.815775 12271 raft_consensus.cc:385] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:47.815804 12271 raft_consensus.cc:740] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748516283d5740ecb52d9433a34bbe70, State: Initialized, Role: FOLLOWER
I20260319 16:37:47.815733 12272 raft_consensus.cc:359] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.815851 12272 raft_consensus.cc:385] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:47.815878 12272 raft_consensus.cc:740] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0809967cb7364f3e92ddd67c63a02e00, State: Initialized, Role: FOLLOWER
I20260319 16:37:47.815894 12271 consensus_queue.cc:260] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.815963 12272 consensus_queue.cc:260] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.816128 12271 ts_tablet_manager.cc:1434] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:47.816192 12272 ts_tablet_manager.cc:1434] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:47.816242 12125 heartbeater.cc:499] Master 127.24.187.254:38903 was elected leader, sending a full tablet report...
I20260319 16:37:47.816452 12256 heartbeater.cc:499] Master 127.24.187.254:38903 was elected leader, sending a full tablet report...
W20260319 16:37:47.915606 12126 tablet.cc:2379] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:47.925405 12276 raft_consensus.cc:493] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:47.925515 12276 raft_consensus.cc:515] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.925836 12276 leader_election.cc:290] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0809967cb7364f3e92ddd67c63a02e00 (127.24.187.195:37631), 748516283d5740ecb52d9433a34bbe70 (127.24.187.194:45659)
I20260319 16:37:47.928356 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.925984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:37631 (local address 127.24.187.193:44377)
0319 16:37:47.926043 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.926055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.926095 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.926283 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.926284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.926286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.926495 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.926498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.927373 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.927374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.928113 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.928115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.928204 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.928210 (+ 6us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.928241 (+ 31us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.928282 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260319 16:37:47.928736 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.926116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:44377 (local address 127.24.187.195:37631)
0319 16:37:47.926166 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.926168 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.926174 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:37:47.926199 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.926200 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.926210 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.926259 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.926648 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.927247 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.928213 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.928627 (+ 414us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.928636 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.928659 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.928687 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260319 16:37:47.928887 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.926008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:45659 (local address 127.24.187.193:40823)
0319 16:37:47.926175 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.926184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:47.926241 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:47.926387 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:47.926387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:47.926401 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:47.927032 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.927035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.927811 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.927812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:47.928702 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:47.928704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.928754 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.928757 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:47.928776 (+ 19us) client_negotiation.cc:240] Negotiation successful
0319 16:37:47.928811 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260319 16:37:47.929010 12210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bafd105e5354a02bc89740f6334cfae" candidate_uuid: "cda32e496dc84ff4b988b01d914002dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0809967cb7364f3e92ddd67c63a02e00" is_pre_election: true
I20260319 16:37:47.929160 12210 raft_consensus.cc:2468] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda32e496dc84ff4b988b01d914002dc in term 0.
I20260319 16:37:47.929320 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.926089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:40823 (local address 127.24.187.194:45659)
0319 16:37:47.926130 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.926132 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.926245 (+ 113us) server_negotiation.cc:408] Connection header received
0319 16:37:47.926310 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.926312 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.926322 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.926371 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:47.927131 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.927695 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.928779 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.929201 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.929210 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.929233 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.929266 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:37:47.929431 11883 leader_election.cc:304] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0809967cb7364f3e92ddd67c63a02e00, cda32e496dc84ff4b988b01d914002dc; no voters:
I20260319 16:37:47.929553 12276 raft_consensus.cc:2804] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:47.929632 12276 raft_consensus.cc:493] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:47.929654 12276 raft_consensus.cc:3060] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:47.929778 12079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bafd105e5354a02bc89740f6334cfae" candidate_uuid: "cda32e496dc84ff4b988b01d914002dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748516283d5740ecb52d9433a34bbe70" is_pre_election: true
I20260319 16:37:47.929930 12079 raft_consensus.cc:2468] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda32e496dc84ff4b988b01d914002dc in term 0.
I20260319 16:37:47.930501 12276 raft_consensus.cc:515] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.930625 12276 leader_election.cc:290] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [CANDIDATE]: Term 1 election: Requested vote from peers 0809967cb7364f3e92ddd67c63a02e00 (127.24.187.195:37631), 748516283d5740ecb52d9433a34bbe70 (127.24.187.194:45659)
I20260319 16:37:47.930821 12210 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bafd105e5354a02bc89740f6334cfae" candidate_uuid: "cda32e496dc84ff4b988b01d914002dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0809967cb7364f3e92ddd67c63a02e00"
I20260319 16:37:47.930830 12079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bafd105e5354a02bc89740f6334cfae" candidate_uuid: "cda32e496dc84ff4b988b01d914002dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748516283d5740ecb52d9433a34bbe70"
I20260319 16:37:47.930902 12079 raft_consensus.cc:3060] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:47.930897 12210 raft_consensus.cc:3060] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:47.931567 12079 raft_consensus.cc:2468] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda32e496dc84ff4b988b01d914002dc in term 1.
I20260319 16:37:47.931743 11881 leader_election.cc:304] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 748516283d5740ecb52d9433a34bbe70, cda32e496dc84ff4b988b01d914002dc; no voters:
I20260319 16:37:47.931839 12276 raft_consensus.cc:2804] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:47.931953 12210 raft_consensus.cc:2468] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda32e496dc84ff4b988b01d914002dc in term 1.
I20260319 16:37:47.932008 12276 raft_consensus.cc:697] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 1 LEADER]: Becoming Leader. State: Replica: cda32e496dc84ff4b988b01d914002dc, State: Running, Role: LEADER
I20260319 16:37:47.932125 12276 consensus_queue.cc:237] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } }
I20260319 16:37:47.932934 11805 catalog_manager.cc:5671] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc reported cstate change: term changed from 0 to 1, leader changed from <none> to cda32e496dc84ff4b988b01d914002dc (127.24.187.193). New cstate: current_term: 1 leader_uuid: "cda32e496dc84ff4b988b01d914002dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cda32e496dc84ff4b988b01d914002dc" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 42797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 } health_report { overall_health: UNKNOWN } } }
I20260319 16:37:47.958261 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.956141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.24.187.193:42797)
0319 16:37:47.956198 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:47.956200 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:47.956207 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:37:47.956250 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:47.956251 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:47.956268 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:47.956316 (+ 48us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:47.956578 (+ 262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.957128 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:47.957750 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:47.957898 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:47.957939 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:47.958027 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:37:47.958096 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:47.958159 (+ 63us) server_negotiation.cc:300] Negotiation successful
0319 16:37:47.958187 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260319 16:37:47.959830 12210 raft_consensus.cc:1275] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 1 FOLLOWER]: Refusing update from remote peer cda32e496dc84ff4b988b01d914002dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:47.959872 12079 raft_consensus.cc:1275] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 1 FOLLOWER]: Refusing update from remote peer cda32e496dc84ff4b988b01d914002dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:47.960117 12276 consensus_queue.cc:1048] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0809967cb7364f3e92ddd67c63a02e00" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 37631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:47.960242 12280 consensus_queue.cc:1048] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "748516283d5740ecb52d9433a34bbe70" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:47.962356 12283 mvcc.cc:204] Tried to move back new op lower bound from 7266051145561108480 to 7266051145450541056. Current Snapshot: MvccSnapshot[applied={T|T < 7266051145561108480}]
I20260319 16:37:47.963063 12285 mvcc.cc:204] Tried to move back new op lower bound from 7266051145561108480 to 7266051145450541056. Current Snapshot: MvccSnapshot[applied={T|T < 7266051145561108480 or (T in {7266051145561108480})}]
I20260319 16:37:47.963248 12284 mvcc.cc:204] Tried to move back new op lower bound from 7266051145561108480 to 7266051145450541056. Current Snapshot: MvccSnapshot[applied={T|T < 7266051145561108480}]
I20260319 16:37:47.965327 11804 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:47.965435 11804 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:47.966153 11804 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 2d6ccfadcc64487daa95a79c5a213ef9: Sending DeleteTablet for 3 replicas of tablet 1bafd105e5354a02bc89740f6334cfae
I20260319 16:37:47.966421 12190 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bafd105e5354a02bc89740f6334cfae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:47 UTC) from {username='slave'} at 127.0.0.1:58914
I20260319 16:37:47.966485 12059 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bafd105e5354a02bc89740f6334cfae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:47 UTC) from {username='slave'} at 127.0.0.1:56762
I20260319 16:37:47.966769 12293 tablet_replica.cc:333] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: stopping tablet replica
I20260319 16:37:47.966894 11928 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bafd105e5354a02bc89740f6334cfae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:47 UTC) from {username='slave'} at 127.0.0.1:38534
I20260319 16:37:47.967087 12293 raft_consensus.cc:2243] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:47.967099 12294 tablet_replica.cc:333] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc: stopping tablet replica
I20260319 16:37:47.967113 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11866
I20260319 16:37:47.967203 12293 raft_consensus.cc:2272] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:47.967279 12294 raft_consensus.cc:2243] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:37:47.967449 12294 raft_consensus.cc:2272] T 1bafd105e5354a02bc89740f6334cfae P cda32e496dc84ff4b988b01d914002dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:47.967648 12293 ts_tablet_manager.cc:1916] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:47.968945 12293 ts_tablet_manager.cc:1929] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:47.969002 12293 log.cc:1199] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/1bafd105e5354a02bc89740f6334cfae
I20260319 16:37:47.969208 12293 ts_tablet_manager.cc:1950] T 1bafd105e5354a02bc89740f6334cfae P 748516283d5740ecb52d9433a34bbe70: Deleting consensus metadata
I20260319 16:37:47.969538 11790 catalog_manager.cc:5002] TS 748516283d5740ecb52d9433a34bbe70 (127.24.187.194:45659): tablet 1bafd105e5354a02bc89740f6334cfae (table test-table [id=0577f4ac091e43a6b262c943a89e95f0]) successfully deleted
I20260319 16:37:47.973326 12292 tablet_replica.cc:333] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: stopping tablet replica
I20260319 16:37:47.973431 12292 raft_consensus.cc:2243] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:47.973529 12292 raft_consensus.cc:2272] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:47.973941 12292 ts_tablet_manager.cc:1916] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260319 16:37:47.974231 11791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:42797 (error 108)
I20260319 16:37:47.974515 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11997
I20260319 16:37:47.974735 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:47.974616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:42797 (local address 127.0.0.1:38548)
0319 16:37:47.974668 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:47.974704 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:42797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260319 16:37:47.975252 11791 catalog_manager.cc:4729] TS cda32e496dc84ff4b988b01d914002dc (127.24.187.193:42797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bafd105e5354a02bc89740f6334cfae: Network error: Client connection negotiation failed: client connection to 127.24.187.193:42797: connect: Connection refused (error 111)
I20260319 16:37:47.975427 12292 ts_tablet_manager.cc:1929] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:47.975518 12292 log.cc:1199] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/1bafd105e5354a02bc89740f6334cfae
I20260319 16:37:47.975783 12292 ts_tablet_manager.cc:1950] T 1bafd105e5354a02bc89740f6334cfae P 0809967cb7364f3e92ddd67c63a02e00: Deleting consensus metadata
I20260319 16:37:47.976225 11792 catalog_manager.cc:5002] TS 0809967cb7364f3e92ddd67c63a02e00 (127.24.187.195:37631): tablet 1bafd105e5354a02bc89740f6334cfae (table test-table [id=0577f4ac091e43a6b262c943a89e95f0]) successfully deleted
I20260319 16:37:47.979230 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 12128
I20260319 16:37:47.984467 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 11775
2026-03-19T16:37:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (742 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:37:48 dist-test-slave-svn8 krb5kdc[12298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:37:48 dist-test-slave-svn8 krb5kdc[12298](info): set up 2 sockets
Mar 19 16:37:48 dist-test-slave-svn8 krb5kdc[12298](info): commencing operation
krb5kdc: starting...
W20260319 16:37:49.826102 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.823s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:37:49 dist-test-slave-svn8 krb5kdc[12298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:37:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:49.968798 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39623
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:50.043656 12314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:50.043835 12314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:50.043864 12314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:50.045377 12314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:50.045423 12314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:50.045440 12314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:50.045456 12314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:50.045471 12314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:50.047148 12314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:50.047386 12314 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:50.047549 12314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:50.049968 12320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:50.049983 12322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:50.049973 12319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:50.050364 12314 server_base.cc:1060] running on GCE node
I20260319 16:37:50.050537 12314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:50.050750 12314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:50.051880 12314 hybrid_clock.cc:648] HybridClock initialized: now 1773938270051864 us; error 27 us; skew 500 ppm
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938270, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:50.054292 12314 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:37:50.054795 12314 webserver.cc:492] Webserver started at http://127.24.187.254:46823/ using document root <none> and password file <none>
I20260319 16:37:50.054961 12314 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:50.054996 12314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:50.055094 12314 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:50.056002 12314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "1382a6d977ee46f786aa68c8cddfb91f"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "b4d59385ffe6870ca7fe9c43c7d91849"
server_key_iv: "746a52b8536200bfef6776bb379a2f15"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.056296 12314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "1382a6d977ee46f786aa68c8cddfb91f"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "b4d59385ffe6870ca7fe9c43c7d91849"
server_key_iv: "746a52b8536200bfef6776bb379a2f15"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.057369 12314 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:50.058015 12329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.058209 12314 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.058269 12314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "1382a6d977ee46f786aa68c8cddfb91f"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "b4d59385ffe6870ca7fe9c43c7d91849"
server_key_iv: "746a52b8536200bfef6776bb379a2f15"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.058334 12314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:50.071611 12314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:50.077517 12314 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:50.077674 12314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:50.081439 12314 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39623
I20260319 16:37:50.081485 12382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39623 every 8 connection(s)
I20260319 16:37:50.081807 12314 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:50.082383 12383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:50.083323 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 12314
I20260319 16:37:50.083451 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:50.083666 25327 external_mini_cluster.cc:1468] Setting key 9effb9afd5ccad268dd4b669edf33263
I20260319 16:37:50.085260 12383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f: Bootstrap starting.
I20260319 16:37:50.085816 12383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:50.086063 12383 log.cc:826] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:50.086757 12383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f: No bootstrap required, opened a new log
I20260319 16:37:50.087942 12383 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1382a6d977ee46f786aa68c8cddfb91f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39623 } }
I20260319 16:37:50.088060 12383 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:50.088088 12383 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1382a6d977ee46f786aa68c8cddfb91f, State: Initialized, Role: FOLLOWER
I20260319 16:37:50.088186 12383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1382a6d977ee46f786aa68c8cddfb91f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39623 } }
I20260319 16:37:50.088248 12383 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:50.088275 12383 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:50.088330 12383 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 0 FOLLOWER]: Advancing to term 1
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:50.088877 12383 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1382a6d977ee46f786aa68c8cddfb91f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39623 } }
I20260319 16:37:50.088989 12383 leader_election.cc:304] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1382a6d977ee46f786aa68c8cddfb91f; no voters:
I20260319 16:37:50.089146 12383 leader_election.cc:290] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:50.089195 12388 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:50.089381 12383 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:50.089407 12388 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [term 1 LEADER]: Becoming Leader. State: Replica: 1382a6d977ee46f786aa68c8cddfb91f, State: Running, Role: LEADER
I20260319 16:37:50.089531 12388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1382a6d977ee46f786aa68c8cddfb91f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39623 } }
I20260319 16:37:50.090009 12389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1382a6d977ee46f786aa68c8cddfb91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1382a6d977ee46f786aa68c8cddfb91f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39623 } } }
I20260319 16:37:50.090230 12389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:50.090164 12390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1382a6d977ee46f786aa68c8cddfb91f. Latest consensus state: current_term: 1 leader_uuid: "1382a6d977ee46f786aa68c8cddfb91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1382a6d977ee46f786aa68c8cddfb91f" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39623 } } }
I20260319 16:37:50.090481 12390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:50.090492 12397 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:50.091015 12397 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:50.092695 12397 catalog_manager.cc:1357] Generated new cluster ID: c07c3da6a92b4852bd83aed93979f1c5
I20260319 16:37:50.092749 12397 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:37:50.092861 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.084161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.24.187.254:39623)
0319 16:37:50.084329 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.084340 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.084363 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:37:50.084691 (+ 328us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.084696 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.084733 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.084844 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:50.085619 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.086455 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.087042 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.087296 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.089343 (+ 2047us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:50.089351 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:50.089352 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:50.089372 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:50.091388 (+ 2016us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.091713 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.091715 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.091716 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.091774 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.091948 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.091949 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.091950 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.092176 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:50.092298 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.092532 (+ 234us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.092661 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260319 16:37:50.110013 12397 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:50.110646 12397 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:50.119117 12397 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f: Generated new TSK 0
I20260319 16:37:50.119305 12397 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:50.174996 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39623
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:50.249608 12411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:50.249812 12411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:50.249840 12411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:50.251323 12411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:50.251376 12411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:50.251402 12411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:50.253010 12411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:50.253221 12411 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:50.253407 12411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:50.255785 12416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:50.255810 12419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:50.256057 12411 server_base.cc:1060] running on GCE node
W20260319 16:37:50.255815 12417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:50.256270 12411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:50.256438 12411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:50.257628 12411 hybrid_clock.cc:648] HybridClock initialized: now 1773938270257612 us; error 27 us; skew 500 ppm
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938270, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:50.259862 12411 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:37:50.260303 12411 webserver.cc:492] Webserver started at http://127.24.187.193:42263/ using document root <none> and password file <none>
I20260319 16:37:50.260478 12411 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:50.260509 12411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:50.260581 12411 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:50.261430 12411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "44fa07cff4eb4b2fa7936d9f4f47e345"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "86364e35c51773e325262720a96ea591"
server_key_iv: "c64fb4875268a2eb3346578bc6172b5c"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.261902 12411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "44fa07cff4eb4b2fa7936d9f4f47e345"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "86364e35c51773e325262720a96ea591"
server_key_iv: "c64fb4875268a2eb3346578bc6172b5c"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.263106 12411 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:50.263785 12426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.264024 12411 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:50.264086 12411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "44fa07cff4eb4b2fa7936d9f4f47e345"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "86364e35c51773e325262720a96ea591"
server_key_iv: "c64fb4875268a2eb3346578bc6172b5c"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.264133 12411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:50.275125 12411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:50.276932 12411 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:50.277031 12411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:50.277196 12411 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:50.277449 12411 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:50.277477 12411 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.277498 12411 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:50.277516 12411 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.283442 12411 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:33277
I20260319 16:37:50.283537 12539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:33277 every 8 connection(s)
I20260319 16:37:50.283779 12411 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938270, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:50.289595 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 12411
I20260319 16:37:50.289798 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:37:50.290010 25327 external_mini_cluster.cc:1468] Setting key ac1c641fef3d59c90f0c0d0a83448fbb
I20260319 16:37:50.292683 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.284643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:35725 (local address 127.24.187.254:39623)
0319 16:37:50.284697 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.284699 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.284805 (+ 106us) server_negotiation.cc:408] Connection header received
0319 16:37:50.285307 (+ 502us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.285308 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.285320 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.285379 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:50.286132 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.286658 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.287178 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.287349 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.289581 (+ 2232us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:50.289586 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:50.289586 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:50.289594 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:50.291333 (+ 1739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.291758 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.291758 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.291759 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.291776 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.291990 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.291991 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.291992 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.292098 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:50.292159 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.292483 (+ 324us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.292590 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:37:50.292774 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.284536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39623 (local address 127.24.187.193:35725)
0319 16:37:50.284703 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.284735 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.285218 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.285441 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.285444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.285767 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:50.286041 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.286045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.286742 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.286742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.287102 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.287104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.287209 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.287691 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:50.287706 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:50.289481 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:50.291412 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:50.291419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:50.291421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:50.291688 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:50.291825 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:50.291826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:50.291826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:50.291906 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:50.292161 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:50.292165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:50.292313 (+ 148us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.292427 (+ 114us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.292574 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260319 16:37:50.293358 12540 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39623
I20260319 16:37:50.293452 12540 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:50.293658 12540 heartbeater.cc:507] Master 127.24.187.254:39623 requested a full tablet report, sending...
I20260319 16:37:50.294155 12346 ts_manager.cc:194] Registered new tserver with Master: 44fa07cff4eb4b2fa7936d9f4f47e345 (127.24.187.193:33277)
I20260319 16:37:50.294823 12346 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:35725
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:50.341864 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39623
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:50.417068 12547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:50.417253 12547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:50.417284 12547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:50.418886 12547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:50.418955 12547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:50.418985 12547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:50.420733 12547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:50.420921 12547 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:50.421057 12547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:50.423621 12555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:50.423628 12553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:50.423748 12552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:50.423818 12547 server_base.cc:1060] running on GCE node
I20260319 16:37:50.423982 12547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:50.424158 12547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:50.425333 12547 hybrid_clock.cc:648] HybridClock initialized: now 1773938270425312 us; error 30 us; skew 500 ppm
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938270, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:50.427898 12547 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:37:50.428467 12547 webserver.cc:492] Webserver started at http://127.24.187.194:36243/ using document root <none> and password file <none>
I20260319 16:37:50.428658 12547 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:50.428692 12547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:50.428792 12547 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:50.429697 12547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "37bbd2829f1d4b29b43a935882a8b408"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "e453ef15e66f2c5e1b3be4de39f23f0d"
server_key_iv: "9619e8cabb5eb52adc11c7120d1bfd50"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.430043 12547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "37bbd2829f1d4b29b43a935882a8b408"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "e453ef15e66f2c5e1b3be4de39f23f0d"
server_key_iv: "9619e8cabb5eb52adc11c7120d1bfd50"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.431151 12547 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:50.431761 12562 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.431933 12547 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:37:50.431993 12547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "37bbd2829f1d4b29b43a935882a8b408"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "e453ef15e66f2c5e1b3be4de39f23f0d"
server_key_iv: "9619e8cabb5eb52adc11c7120d1bfd50"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.432054 12547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:50.442778 12547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:50.444722 12547 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:50.444850 12547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:50.445050 12547 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:50.445356 12547 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:50.445387 12547 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.445415 12547 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:50.445432 12547 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.450959 12547 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:41389
I20260319 16:37:50.451073 12675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:41389 every 8 connection(s)
I20260319 16:37:50.451290 12547 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938270, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:50.456815 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 12547
I20260319 16:37:50.456926 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:37:50.457114 25327 external_mini_cluster.cc:1468] Setting key ce79c53fcc4506743111cef413d81527
I20260319 16:37:50.459314 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.451900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:36923 (local address 127.24.187.254:39623)
0319 16:37:50.451970 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.451972 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.452142 (+ 170us) server_negotiation.cc:408] Connection header received
0319 16:37:50.452641 (+ 499us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.452643 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.452655 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.452704 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:50.453427 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.453961 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.454465 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.454632 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.456808 (+ 2176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:50.456812 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:50.456812 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:50.456820 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:50.458206 (+ 1386us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.458567 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.458568 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.458568 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.458592 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.458741 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.458741 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.458742 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.458840 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:50.458899 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.459152 (+ 253us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.459233 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260319 16:37:50.459389 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.451854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39623 (local address 127.24.187.194:36923)
0319 16:37:50.452033 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.452064 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.452555 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.452762 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.452765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.453053 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:50.453335 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.453339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.454036 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.454038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.454394 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.454395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.454509 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.454947 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:50.454962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:50.456722 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:50.458275 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:50.458281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:50.458282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:50.458499 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:50.458647 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:50.458647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:50.458648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:50.458696 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:50.458929 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:50.458934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:50.459039 (+ 105us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.459127 (+ 88us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.459231 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260319 16:37:50.459847 12676 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39623
I20260319 16:37:50.459928 12676 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:50.460094 12676 heartbeater.cc:507] Master 127.24.187.254:39623 requested a full tablet report, sending...
I20260319 16:37:50.460424 12346 ts_manager.cc:194] Registered new tserver with Master: 37bbd2829f1d4b29b43a935882a8b408 (127.24.187.194:41389)
I20260319 16:37:50.460796 12346 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:36923
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:50.505579 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:39623
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:50.581439 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:50.581624 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:50.581655 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:50.583330 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:50.583388 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:50.583428 12683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:50.585127 12683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:39623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:50.585359 12683 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:50.585525 12683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:50.588092 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:50.588088 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:50.588102 12683 server_base.cc:1060] running on GCE node
W20260319 16:37:50.588088 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:50.588519 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:50.588723 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:50.589859 12683 hybrid_clock.cc:648] HybridClock initialized: now 1773938270589838 us; error 29 us; skew 500 ppm
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938270, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:50.592198 12683 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:37:50.592787 12683 webserver.cc:492] Webserver started at http://127.24.187.195:40207/ using document root <none> and password file <none>
I20260319 16:37:50.593006 12683 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:50.593055 12683 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:50.593178 12683 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:50.594143 12683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "68e558bc4dd5494f9d6913d8a508d003"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "1c9cffad0c5534446deafbed8f5a0af8"
server_key_iv: "1c43f656b42c1149607a619b197f1e03"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.594436 12683 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "68e558bc4dd5494f9d6913d8a508d003"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "1c9cffad0c5534446deafbed8f5a0af8"
server_key_iv: "1c43f656b42c1149607a619b197f1e03"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.595615 12683 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:50.596372 12698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.596659 12683 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:50.596724 12683 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "68e558bc4dd5494f9d6913d8a508d003"
format_stamp: "Formatted at 2026-03-19 16:37:50 on dist-test-slave-svn8"
server_key: "1c9cffad0c5534446deafbed8f5a0af8"
server_key_iv: "1c43f656b42c1149607a619b197f1e03"
server_key_version: "encryptionkey@0"
I20260319 16:37:50.596778 12683 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:50.607797 12683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:50.609540 12683 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:50.609644 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:50.609894 12683 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:50.610201 12683 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:50.610235 12683 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.610257 12683 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:50.610268 12683 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:50.615744 12683 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:38023
I20260319 16:37:50.615850 12811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:38023 every 8 connection(s)
I20260319 16:37:50.616073 12683 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:50.620110 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 12683
I20260319 16:37:50.620211 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:37:50.620420 25327 external_mini_cluster.cc:1468] Setting key 36b6d587267f1e6e47c0d1c7a57020d2
Mar 19 16:37:50 dist-test-slave-svn8 krb5kdc[12298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938270, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:50.624601 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.616730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:52261 (local address 127.24.187.254:39623)
0319 16:37:50.616804 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.616806 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.616915 (+ 109us) server_negotiation.cc:408] Connection header received
0319 16:37:50.617436 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.617438 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.617449 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.617536 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:50.618280 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.618824 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.619541 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.619720 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.622120 (+ 2400us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:50.622124 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:50.622125 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:50.622132 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:50.623412 (+ 1280us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.623777 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.623778 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.623778 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.623805 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.624012 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.624013 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.624013 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.624110 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:50.624176 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.624445 (+ 269us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.624525 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260319 16:37:50.624652 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.616678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:39623 (local address 127.24.187.195:52261)
0319 16:37:50.616809 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.616838 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.617324 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.617583 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.617586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.617921 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:50.618189 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.618192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.618928 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.618930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.619439 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.619442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.619555 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.620017 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:50.620033 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:50.622042 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:50.623490 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:50.623495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:50.623497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:50.623688 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:50.623875 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:50.623876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:50.623876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:50.623934 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:50.624177 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:50.624181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:50.624305 (+ 124us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.624395 (+ 90us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.624498 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260319 16:37:50.625075 12812 heartbeater.cc:344] Connected to a master server at 127.24.187.254:39623
I20260319 16:37:50.625162 12812 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:50.625350 12812 heartbeater.cc:507] Master 127.24.187.254:39623 requested a full tablet report, sending...
I20260319 16:37:50.625898 12346 ts_manager.cc:194] Registered new tserver with Master: 68e558bc4dd5494f9d6913d8a508d003 (127.24.187.195:38023)
I20260319 16:37:50.626259 12346 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:52261
I20260319 16:37:50.632112 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:37:50.639163 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.633548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.24.187.254:39623)
0319 16:37:50.633618 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.633620 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.633669 (+ 49us) server_negotiation.cc:408] Connection header received
0319 16:37:50.633750 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.633751 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.633764 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.633817 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:50.634456 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.634934 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.635540 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.635693 (+ 153us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.636753 (+ 1060us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:50.636756 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:50.636756 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:50.636763 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:50.638020 (+ 1257us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.638407 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.638408 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.638409 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.638432 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:50.638645 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:50.638647 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:50.638647 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:50.638752 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:50.638844 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.639045 (+ 201us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.639099 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260319 16:37:50.640851 12346 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:37:50.642149 12346 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:37:50.647629 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.644348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33277 (local address 127.0.0.1:40150)
0319 16:37:50.644891 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.644897 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.644942 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.645113 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.645114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.645116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:50.645348 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.645350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.646675 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.646676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.647318 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.647320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.647380 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.647423 (+ 43us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.647467 (+ 44us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.647507 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":86,"threads_started":1}
I20260319 16:37:50.647789 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.644321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:38023 (local address 127.0.0.1:43084)
0319 16:37:50.644738 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.644749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.644820 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.644984 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.644987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.644989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:50.645266 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.645268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.646643 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.646644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.647315 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.647317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.647383 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.647423 (+ 40us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.647480 (+ 57us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.647538 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":77,"threads_started":1}
I20260319 16:37:50.648147 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.644461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.24.187.195:38023)
0319 16:37:50.644635 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.644642 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.644825 (+ 183us) server_negotiation.cc:408] Connection header received
0319 16:37:50.644889 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.644891 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.644905 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.644965 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:50.645356 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.646547 (+ 1191us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.647423 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.647937 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.647993 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.648026 (+ 33us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.648070 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260319 16:37:50.648514 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.644286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:41389 (local address 127.0.0.1:55992)
0319 16:37:50.644491 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.644549 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.644621 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.645527 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.645527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.645530 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:50.645685 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.645686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.647612 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.647612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.648287 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.648290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.648364 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.648367 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.648403 (+ 36us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.648440 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
I20260319 16:37:50.648741 12746 tablet_service.cc:1511] Processing CreateTablet for tablet 1b75a2bec6084509b958f9f268fb4ee9 (DEFAULT_TABLE table=test-table [id=269becd5148041dfab2bf7a09d5cfa7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:50.648929 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.644584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.24.187.193:33277)
0319 16:37:50.644760 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.644769 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.644953 (+ 184us) server_negotiation.cc:408] Connection header received
0319 16:37:50.645003 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.645006 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.645023 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.645090 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:50.645439 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.646595 (+ 1156us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.648269 (+ 1674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.648770 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.648803 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.648827 (+ 24us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.648859 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260319 16:37:50.649089 12746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b75a2bec6084509b958f9f268fb4ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:50.649364 12474 tablet_service.cc:1511] Processing CreateTablet for tablet 1b75a2bec6084509b958f9f268fb4ee9 (DEFAULT_TABLE table=test-table [id=269becd5148041dfab2bf7a09d5cfa7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:50.649456 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.644483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.24.187.194:41389)
0319 16:37:50.645272 (+ 789us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.645286 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.645298 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:37:50.645341 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.645343 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.645360 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.645427 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:50.645801 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.646550 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.648707 (+ 2157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.649261 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.649303 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.649338 (+ 35us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.649381 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":86,"threads_started":1}
I20260319 16:37:50.649562 12474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b75a2bec6084509b958f9f268fb4ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:50.650034 12610 tablet_service.cc:1511] Processing CreateTablet for tablet 1b75a2bec6084509b958f9f268fb4ee9 (DEFAULT_TABLE table=test-table [id=269becd5148041dfab2bf7a09d5cfa7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:50.650264 12610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b75a2bec6084509b958f9f268fb4ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:50.651667 12829 tablet_bootstrap.cc:492] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Bootstrap starting.
I20260319 16:37:50.652268 12829 tablet_bootstrap.cc:654] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:50.652259 12830 tablet_bootstrap.cc:492] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345: Bootstrap starting.
I20260319 16:37:50.652702 12829 log.cc:826] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:50.652833 12830 tablet_bootstrap.cc:654] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:50.652948 12831 tablet_bootstrap.cc:492] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Bootstrap starting.
I20260319 16:37:50.653164 12830 log.cc:826] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:50.653481 12831 tablet_bootstrap.cc:654] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:50.653577 12829 tablet_bootstrap.cc:492] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: No bootstrap required, opened a new log
I20260319 16:37:50.653645 12829 ts_tablet_manager.cc:1403] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:50.653811 12831 log.cc:826] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:50.653854 12830 tablet_bootstrap.cc:492] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345: No bootstrap required, opened a new log
I20260319 16:37:50.653920 12830 ts_tablet_manager.cc:1403] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:50.654500 12831 tablet_bootstrap.cc:492] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: No bootstrap required, opened a new log
I20260319 16:37:50.654573 12831 ts_tablet_manager.cc:1403] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260319 16:37:50.655125 12829 raft_consensus.cc:359] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.655201 12830 raft_consensus.cc:359] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.655289 12829 raft_consensus.cc:385] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:50.655323 12829 raft_consensus.cc:740] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e558bc4dd5494f9d6913d8a508d003, State: Initialized, Role: FOLLOWER
I20260319 16:37:50.655313 12830 raft_consensus.cc:385] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:50.655352 12830 raft_consensus.cc:740] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fa07cff4eb4b2fa7936d9f4f47e345, State: Initialized, Role: FOLLOWER
I20260319 16:37:50.655417 12829 consensus_queue.cc:260] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.655464 12830 consensus_queue.cc:260] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.655635 12829 ts_tablet_manager.cc:1434] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260319 16:37:50.655748 12830 ts_tablet_manager.cc:1434] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:50.655750 12812 heartbeater.cc:499] Master 127.24.187.254:39623 was elected leader, sending a full tablet report...
I20260319 16:37:50.655861 12540 heartbeater.cc:499] Master 127.24.187.254:39623 was elected leader, sending a full tablet report...
I20260319 16:37:50.656144 12831 raft_consensus.cc:359] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.656279 12831 raft_consensus.cc:385] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:50.656313 12831 raft_consensus.cc:740] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bbd2829f1d4b29b43a935882a8b408, State: Initialized, Role: FOLLOWER
I20260319 16:37:50.656404 12831 consensus_queue.cc:260] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.657044 12676 heartbeater.cc:499] Master 127.24.187.254:39623 was elected leader, sending a full tablet report...
I20260319 16:37:50.657025 12831 ts_tablet_manager.cc:1434] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260319 16:37:50.701838 12677 tablet.cc:2379] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:37:50.784485 12541 tablet.cc:2379] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:50.822002 12836 raft_consensus.cc:493] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:50.822141 12836 raft_consensus.cc:515] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.822768 12836 leader_election.cc:290] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68e558bc4dd5494f9d6913d8a508d003 (127.24.187.195:38023), 37bbd2829f1d4b29b43a935882a8b408 (127.24.187.194:41389)
I20260319 16:37:50.825146 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.822665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:41389 (local address 127.24.187.193:48699)
0319 16:37:50.822784 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.822797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.822870 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.823017 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.823018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.823021 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:50.823298 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.823300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.824226 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.824227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.824900 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.824904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.824989 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.824994 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.825025 (+ 31us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.825061 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260319 16:37:50.825481 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.822665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:38023 (local address 127.24.187.193:36265)
0319 16:37:50.822830 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.822841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:50.822938 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:50.823141 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:50.823142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:50.823144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:50.823384 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.823386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.824292 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.824293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:50.825005 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:50.825008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.825074 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.825078 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:50.825106 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:37:50.825139 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":10,"thread_start_us":80,"threads_started":1}
I20260319 16:37:50.825585 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.822792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:48699 (local address 127.24.187.194:41389)
0319 16:37:50.822847 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.822850 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.822862 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:37:50.822938 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.822940 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.822950 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.823000 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:50.823384 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.824120 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.824988 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.825461 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.825470 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.825496 (+ 26us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.825526 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:37:50.825678 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.822881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:36265 (local address 127.24.187.195:38023)
0319 16:37:50.822928 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.822930 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.822939 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:50.823006 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.823008 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.823016 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.823065 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:50.823471 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.824185 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.825094 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.825552 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.825562 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.825587 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.825618 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260319 16:37:50.825992 12630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b75a2bec6084509b958f9f268fb4ee9" candidate_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bbd2829f1d4b29b43a935882a8b408" is_pre_election: true
I20260319 16:37:50.826125 12766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b75a2bec6084509b958f9f268fb4ee9" candidate_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e558bc4dd5494f9d6913d8a508d003" is_pre_election: true
I20260319 16:37:50.826171 12630 raft_consensus.cc:2468] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fa07cff4eb4b2fa7936d9f4f47e345 in term 0.
I20260319 16:37:50.826256 12766 raft_consensus.cc:2468] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fa07cff4eb4b2fa7936d9f4f47e345 in term 0.
I20260319 16:37:50.826400 12430 leader_election.cc:304] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bbd2829f1d4b29b43a935882a8b408, 44fa07cff4eb4b2fa7936d9f4f47e345; no voters:
I20260319 16:37:50.826666 12836 raft_consensus.cc:2804] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:50.826728 12836 raft_consensus.cc:493] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:50.826746 12836 raft_consensus.cc:3060] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:50.827698 12836 raft_consensus.cc:515] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.828148 12836 leader_election.cc:290] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [CANDIDATE]: Term 1 election: Requested vote from peers 68e558bc4dd5494f9d6913d8a508d003 (127.24.187.195:38023), 37bbd2829f1d4b29b43a935882a8b408 (127.24.187.194:41389)
I20260319 16:37:50.828207 12766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b75a2bec6084509b958f9f268fb4ee9" candidate_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e558bc4dd5494f9d6913d8a508d003"
I20260319 16:37:50.828203 12630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b75a2bec6084509b958f9f268fb4ee9" candidate_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bbd2829f1d4b29b43a935882a8b408"
I20260319 16:37:50.828282 12766 raft_consensus.cc:3060] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:50.828331 12630 raft_consensus.cc:3060] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:50.829049 12766 raft_consensus.cc:2468] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fa07cff4eb4b2fa7936d9f4f47e345 in term 1.
I20260319 16:37:50.829061 12630 raft_consensus.cc:2468] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fa07cff4eb4b2fa7936d9f4f47e345 in term 1.
I20260319 16:37:50.829200 12430 leader_election.cc:304] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bbd2829f1d4b29b43a935882a8b408, 44fa07cff4eb4b2fa7936d9f4f47e345; no voters:
I20260319 16:37:50.829288 12836 raft_consensus.cc:2804] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:50.829412 12836 raft_consensus.cc:697] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [term 1 LEADER]: Becoming Leader. State: Replica: 44fa07cff4eb4b2fa7936d9f4f47e345, State: Running, Role: LEADER
I20260319 16:37:50.829511 12836 consensus_queue.cc:237] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } }
I20260319 16:37:50.830264 12344 catalog_manager.cc:5671] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44fa07cff4eb4b2fa7936d9f4f47e345 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44fa07cff4eb4b2fa7936d9f4f47e345" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 33277 } health_report { overall_health: HEALTHY } } }
W20260319 16:37:50.866636 12813 tablet.cc:2379] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:50.876353 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.874237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.24.187.193:33277)
0319 16:37:50.874308 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:50.874310 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:50.874319 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:50.874372 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:50.874373 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:50.874390 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:50.874438 (+ 48us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:50.874666 (+ 228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.875148 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:50.875714 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:50.875860 (+ 146us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:50.876020 (+ 160us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:50.876108 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:37:50.876182 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:50.876266 (+ 84us) server_negotiation.cc:300] Negotiation successful
0319 16:37:50.876273 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260319 16:37:50.878113 12766 raft_consensus.cc:1275] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 1 FOLLOWER]: Refusing update from remote peer 44fa07cff4eb4b2fa7936d9f4f47e345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:50.878113 12630 raft_consensus.cc:1275] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 1 FOLLOWER]: Refusing update from remote peer 44fa07cff4eb4b2fa7936d9f4f47e345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:50.878466 12836 consensus_queue.cc:1048] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bbd2829f1d4b29b43a935882a8b408" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:50.878608 12840 consensus_queue.cc:1048] T 1b75a2bec6084509b958f9f268fb4ee9 P 44fa07cff4eb4b2fa7936d9f4f47e345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68e558bc4dd5494f9d6913d8a508d003" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:50.880877 12843 mvcc.cc:204] Tried to move back new op lower bound from 7266051157513752576 to 7266051157318197248. Current Snapshot: MvccSnapshot[applied={T|T < 7266051157513752576}]
I20260319 16:37:50.881425 12844 mvcc.cc:204] Tried to move back new op lower bound from 7266051157513752576 to 7266051157318197248. Current Snapshot: MvccSnapshot[applied={T|T < 7266051157513752576}]
I20260319 16:37:50.881907 12845 mvcc.cc:204] Tried to move back new op lower bound from 7266051157513752576 to 7266051157318197248. Current Snapshot: MvccSnapshot[applied={T|T < 7266051157513752576 or (T in {7266051157513752576})}]
I20260319 16:37:50.884042 12344 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:50.884127 12344 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:50.884824 12344 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 1382a6d977ee46f786aa68c8cddfb91f: Sending DeleteTablet for 3 replicas of tablet 1b75a2bec6084509b958f9f268fb4ee9
I20260319 16:37:50.885105 12610 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b75a2bec6084509b958f9f268fb4ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:50 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:55992
I20260319 16:37:50.885104 12474 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b75a2bec6084509b958f9f268fb4ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:50 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:40150
I20260319 16:37:50.885191 12746 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b75a2bec6084509b958f9f268fb4ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:50 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:43084
I20260319 16:37:50.885351 12854 tablet_replica.cc:333] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: stopping tablet replica
I20260319 16:37:50.885453 12855 tablet_replica.cc:333] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: stopping tablet replica
I20260319 16:37:50.885460 12854 raft_consensus.cc:2243] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:50.885529 12855 raft_consensus.cc:2243] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:50.885605 12854 raft_consensus.cc:2272] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:50.885613 12855 raft_consensus.cc:2272] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:50.886031 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 12411
I20260319 16:37:50.886718 12855 ts_tablet_manager.cc:1916] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:50.888398 12855 ts_tablet_manager.cc:1929] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:50.888471 12855 log.cc:1199] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/1b75a2bec6084509b958f9f268fb4ee9
I20260319 16:37:50.888600 12854 ts_tablet_manager.cc:1916] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:50.888747 12855 ts_tablet_manager.cc:1950] T 1b75a2bec6084509b958f9f268fb4ee9 P 68e558bc4dd5494f9d6913d8a508d003: Deleting consensus metadata
I20260319 16:37:50.889133 12332 catalog_manager.cc:5002] TS 68e558bc4dd5494f9d6913d8a508d003 (127.24.187.195:38023): tablet 1b75a2bec6084509b958f9f268fb4ee9 (table test-table [id=269becd5148041dfab2bf7a09d5cfa7b]) successfully deleted
I20260319 16:37:50.890027 12854 ts_tablet_manager.cc:1929] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:50.890089 12854 log.cc:1199] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/1b75a2bec6084509b958f9f268fb4ee9
I20260319 16:37:50.890355 12854 ts_tablet_manager.cc:1950] T 1b75a2bec6084509b958f9f268fb4ee9 P 37bbd2829f1d4b29b43a935882a8b408: Deleting consensus metadata
I20260319 16:37:50.890755 12333 catalog_manager.cc:5002] TS 37bbd2829f1d4b29b43a935882a8b408 (127.24.187.194:41389): tablet 1b75a2bec6084509b958f9f268fb4ee9 (table test-table [id=269becd5148041dfab2bf7a09d5cfa7b]) successfully deleted
W20260319 16:37:50.892202 12330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:33277 (error 108)
I20260319 16:37:50.892422 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 12547
I20260319 16:37:50.892961 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:50.892674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:33277 (local address 127.0.0.1:40180)
0319 16:37:50.892893 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:50.892931 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
W20260319 16:37:50.893064 12330 catalog_manager.cc:4729] TS 44fa07cff4eb4b2fa7936d9f4f47e345 (127.24.187.193:33277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b75a2bec6084509b958f9f268fb4ee9: Network error: Client connection negotiation failed: client connection to 127.24.187.193:33277: connect: Connection refused (error 111)
I20260319 16:37:50.897781 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 12683
I20260319 16:37:50.902611 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 12314
2026-03-19T16:37:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2919 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-03-19T16:37:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:50Z Disabled control of system clock
I20260319 16:37:50.929248 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33813
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:51.003989 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:51.004177 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:51.004199 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:51.005695 12860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:51.005767 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:51.005779 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:51.005791 12860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:51.005801 12860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:51.007782 12860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:33813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:51.007999 12860 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:51.008184 12860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:51.010816 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:51.010789 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:51.010854 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:51.011561 12860 server_base.cc:1060] running on GCE node
I20260319 16:37:51.011807 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:51.012032 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:51.013238 12860 hybrid_clock.cc:648] HybridClock initialized: now 1773938271013182 us; error 27 us; skew 500 ppm
I20260319 16:37:51.014456 12860 webserver.cc:492] Webserver started at http://127.24.187.254:42583/ using document root <none> and password file <none>
I20260319 16:37:51.014650 12860 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:51.014693 12860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:51.014796 12860 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:51.015699 12860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "6e7921859f284cc5bf1cc2ebc7130ba1"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "6d8f5260f661b800dcaa8be3aff0f9b9"
server_key_iv: "04a44b0eff6c54bb9279972f717ecfda"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.015982 12860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "6e7921859f284cc5bf1cc2ebc7130ba1"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "6d8f5260f661b800dcaa8be3aff0f9b9"
server_key_iv: "04a44b0eff6c54bb9279972f717ecfda"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.017050 12860 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:51.017835 12874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.018021 12860 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:37:51.018085 12860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "6e7921859f284cc5bf1cc2ebc7130ba1"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "6d8f5260f661b800dcaa8be3aff0f9b9"
server_key_iv: "04a44b0eff6c54bb9279972f717ecfda"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.018147 12860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:51.029438 12860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:51.029693 12860 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:51.029879 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:51.033880 12860 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:33813
I20260319 16:37:51.033908 12926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:33813 every 8 connection(s)
I20260319 16:37:51.034216 12860 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:51.034864 12927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:51.037176 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1: Bootstrap starting.
I20260319 16:37:51.037889 12927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:51.038125 12927 log.cc:826] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:51.038817 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1: No bootstrap required, opened a new log
I20260319 16:37:51.040091 12927 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33813 } }
I20260319 16:37:51.040212 12927 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:51.040231 12927 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7921859f284cc5bf1cc2ebc7130ba1, State: Initialized, Role: FOLLOWER
I20260319 16:37:51.040369 12927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33813 } }
I20260319 16:37:51.040441 12927 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:51.040460 12927 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:51.040485 12927 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:51.040980 12927 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33813 } }
I20260319 16:37:51.041074 12927 leader_election.cc:304] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e7921859f284cc5bf1cc2ebc7130ba1; no voters:
I20260319 16:37:51.041198 12927 leader_election.cc:290] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:51.041256 12930 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:51.041394 12930 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7921859f284cc5bf1cc2ebc7130ba1, State: Running, Role: LEADER
I20260319 16:37:51.041420 12927 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:51.041474 12930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33813 } }
I20260319 16:37:51.041815 12932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7921859f284cc5bf1cc2ebc7130ba1. Latest consensus state: current_term: 1 leader_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33813 } } }
I20260319 16:37:51.041795 12931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7921859f284cc5bf1cc2ebc7130ba1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33813 } } }
I20260319 16:37:51.041878 12932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:51.041885 12931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:51.042097 12936 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:51.042537 12936 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:51.043176 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 12860
I20260319 16:37:51.043278 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:51.043455 25327 external_mini_cluster.cc:1468] Setting key 47a5784adc4b922af680a1c985dad393
I20260319 16:37:51.044261 12936 catalog_manager.cc:1357] Generated new cluster ID: 55cf0d04949e4117a02201d91cbc3337
I20260319 16:37:51.044301 12936 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:37:51.048491 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.043895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.24.187.254:33813)
0319 16:37:51.044070 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.044075 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.044454 (+ 379us) server_negotiation.cc:408] Connection header received
0319 16:37:51.044564 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.044569 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.044650 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.044765 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:51.045229 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.046360 (+ 1131us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.047147 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.047355 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.047884 (+ 529us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:51.047911 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:51.047911 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:51.048120 (+ 209us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:51.048151 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:51.048159 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:51.048217 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.048298 (+ 81us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.048348 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260319 16:37:51.062870 12936 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:51.063400 12936 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:51.071904 12936 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1: Generated new TSK 0
I20260319 16:37:51.072077 12936 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:37:51.081509 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33813
--builtin_ntp_servers=127.24.187.212:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:51.157462 12951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:51.157626 12951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:51.157647 12951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:51.159127 12951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:51.159173 12951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:51.159198 12951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:51.160722 12951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:51.160907 12951 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:51.161119 12951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:51.163586 12956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:51.163610 12959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:51.163645 12957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:51.163834 12951 server_base.cc:1060] running on GCE node
I20260319 16:37:51.163972 12951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:51.164163 12951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:51.165288 12951 hybrid_clock.cc:648] HybridClock initialized: now 1773938271165244 us; error 55 us; skew 500 ppm
I20260319 16:37:51.166420 12951 webserver.cc:492] Webserver started at http://127.24.187.193:34137/ using document root <none> and password file <none>
I20260319 16:37:51.166596 12951 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:51.166636 12951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:51.166739 12951 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:51.167676 12951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "73d7282f85db431e922707c8215f0b86"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "ee101830b14d61b8ad206ad42f530103"
server_key_iv: "6c6e05e9ef3aeb849918467a7e195b8e"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.168006 12951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "73d7282f85db431e922707c8215f0b86"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "ee101830b14d61b8ad206ad42f530103"
server_key_iv: "6c6e05e9ef3aeb849918467a7e195b8e"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.169121 12951 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:37:51.169849 12965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.170034 12951 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:51.170091 12951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "73d7282f85db431e922707c8215f0b86"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "ee101830b14d61b8ad206ad42f530103"
server_key_iv: "6c6e05e9ef3aeb849918467a7e195b8e"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.170152 12951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:51.201915 12951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:51.202157 12951 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:51.202267 12951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:51.202477 12951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:51.202795 12951 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:51.202826 12951 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.202854 12951 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:51.202872 12951 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.208262 12951 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:36907
I20260319 16:37:51.208320 13078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:36907 every 8 connection(s)
I20260319 16:37:51.208599 12951 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:37:51.212433 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.209245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:33463 (local address 127.24.187.254:33813)
0319 16:37:51.209306 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.209308 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.209479 (+ 171us) server_negotiation.cc:408] Connection header received
0319 16:37:51.209982 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.209984 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.210000 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.210048 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:51.210499 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.210988 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.211511 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.211679 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.212166 (+ 487us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:51.212169 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:51.212170 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:51.212174 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:51.212190 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:51.212193 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:51.212234 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.212312 (+ 78us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.212358 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260319 16:37:51.212524 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.209202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:33813 (local address 127.24.187.193:33463)
0319 16:37:51.209378 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.209405 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.209892 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.210099 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.210104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.210112 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:51.210414 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.210416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.211067 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.211068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.211435 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.211437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.211544 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.212057 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:51.212057 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:51.212075 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:51.212091 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:51.212242 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:51.212246 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:51.212249 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.212311 (+ 62us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.212381 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260319 16:37:51.213063 13079 heartbeater.cc:344] Connected to a master server at 127.24.187.254:33813
I20260319 16:37:51.213158 13079 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:51.213323 13079 heartbeater.cc:507] Master 127.24.187.254:33813 requested a full tablet report, sending...
I20260319 16:37:51.213785 12891 ts_manager.cc:194] Registered new tserver with Master: 73d7282f85db431e922707c8215f0b86 (127.24.187.193:36907)
I20260319 16:37:51.214396 12891 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.193:33463
I20260319 16:37:51.216292 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 12951
I20260319 16:37:51.216378 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:37:51.216555 25327 external_mini_cluster.cc:1468] Setting key c43a321a9b674b92870a40fe05792b29
I20260319 16:37:51.217504 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33813
--builtin_ntp_servers=127.24.187.212:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:51.292279 13082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:51.292456 13082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:51.292484 13082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:51.294006 13082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:51.294059 13082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:51.294091 13082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:51.295652 13082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:51.295881 13082 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:51.296092 13082 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:51.298476 13088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:51.298466 13087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:51.298466 13090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:51.298965 13082 server_base.cc:1060] running on GCE node
I20260319 16:37:51.299134 13082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:51.299343 13082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:51.300501 13082 hybrid_clock.cc:648] HybridClock initialized: now 1773938271300479 us; error 35 us; skew 500 ppm
I20260319 16:37:51.301672 13082 webserver.cc:492] Webserver started at http://127.24.187.194:35853/ using document root <none> and password file <none>
I20260319 16:37:51.301863 13082 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:51.301916 13082 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:51.301996 13082 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:51.302886 13082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "695d99feb7a0418485a330213f5c9d9e"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "f63cd3357f84c631388f7812250c3f28"
server_key_iv: "4aca54ca5dae9ef9af7adff35dd81162"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.303165 13082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "695d99feb7a0418485a330213f5c9d9e"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "f63cd3357f84c631388f7812250c3f28"
server_key_iv: "4aca54ca5dae9ef9af7adff35dd81162"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.304327 13082 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:37:51.305126 13096 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.305321 13082 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:51.305389 13082 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "695d99feb7a0418485a330213f5c9d9e"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "f63cd3357f84c631388f7812250c3f28"
server_key_iv: "4aca54ca5dae9ef9af7adff35dd81162"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.305445 13082 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:51.325551 13082 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:51.325816 13082 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:51.325907 13082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:51.326092 13082 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:51.326403 13082 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:51.326438 13082 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.326459 13082 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:51.326470 13082 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.331669 13082 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:38105
I20260319 16:37:51.331756 13209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:38105 every 8 connection(s)
I20260319 16:37:51.332013 13082 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:37:51.335917 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.332678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:51097 (local address 127.24.187.254:33813)
0319 16:37:51.332739 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.332741 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.332912 (+ 171us) server_negotiation.cc:408] Connection header received
0319 16:37:51.333398 (+ 486us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.333400 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.333416 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.333468 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:51.333939 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.334450 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.334988 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.335142 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.335654 (+ 512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:51.335658 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:51.335659 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:51.335664 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:51.335679 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:51.335682 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:51.335725 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.335799 (+ 74us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.335839 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260319 16:37:51.336017 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.332638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:33813 (local address 127.24.187.194:51097)
0319 16:37:51.332819 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.332843 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.333311 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.333519 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.333523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.333530 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:51.333860 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.333863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.334529 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.334530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.334916 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.334917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.335029 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.335541 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:51.335542 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:51.335559 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:51.335575 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:51.335731 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:51.335735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:51.335737 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.335800 (+ 63us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.335873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260319 16:37:51.336558 13210 heartbeater.cc:344] Connected to a master server at 127.24.187.254:33813
I20260319 16:37:51.336648 13210 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:51.336814 13210 heartbeater.cc:507] Master 127.24.187.254:33813 requested a full tablet report, sending...
I20260319 16:37:51.337141 12891 ts_manager.cc:194] Registered new tserver with Master: 695d99feb7a0418485a330213f5c9d9e (127.24.187.194:38105)
I20260319 16:37:51.337502 12891 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.194:51097
I20260319 16:37:51.342154 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 13082
I20260319 16:37:51.342263 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:37:51.342479 25327 external_mini_cluster.cc:1468] Setting key dc16f91f55aeec1b12a552380f261502
I20260319 16:37:51.343449 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33813
--builtin_ntp_servers=127.24.187.212:43505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:51.417950 13213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:51.418128 13213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:51.418148 13213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:51.419561 13213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:51.419611 13213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:51.419636 13213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:51.421150 13213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:43505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:33813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:51.421358 13213 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:51.421553 13213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:51.424054 13219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:51.424054 13221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:51.424211 13213 server_base.cc:1060] running on GCE node
W20260319 16:37:51.424093 13218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:51.424501 13213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:51.424695 13213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:51.425779 13213 hybrid_clock.cc:648] HybridClock initialized: now 1773938271425767 us; error 26 us; skew 500 ppm
I20260319 16:37:51.426851 13213 webserver.cc:492] Webserver started at http://127.24.187.195:40871/ using document root <none> and password file <none>
I20260319 16:37:51.427023 13213 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:51.427064 13213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:51.427167 13213 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:51.428077 13213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "30aea4ee124446718e27a95d1e2345ff"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "c0709a4ac739eb6ddd66c5f1587ea038"
server_key_iv: "465480878e82d00eb3b4a6a7e3463e2f"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.428364 13213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "30aea4ee124446718e27a95d1e2345ff"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "c0709a4ac739eb6ddd66c5f1587ea038"
server_key_iv: "465480878e82d00eb3b4a6a7e3463e2f"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.429432 13213 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:37:51.430269 13227 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.430475 13213 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:37:51.430552 13213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "30aea4ee124446718e27a95d1e2345ff"
format_stamp: "Formatted at 2026-03-19 16:37:51 on dist-test-slave-svn8"
server_key: "c0709a4ac739eb6ddd66c5f1587ea038"
server_key_iv: "465480878e82d00eb3b4a6a7e3463e2f"
server_key_version: "encryptionkey@0"
I20260319 16:37:51.430622 13213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:51.437613 13213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:51.437853 13213 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:51.437966 13213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:51.438148 13213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:51.438448 13213 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:51.438490 13213 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.438521 13213 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:51.438548 13213 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:51.444583 13213 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:35357
I20260319 16:37:51.444644 13340 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:35357 every 8 connection(s)
I20260319 16:37:51.444916 13213 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:51.448055 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 13213
I20260319 16:37:51.448165 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:37:51.448350 25327 external_mini_cluster.cc:1468] Setting key ea5ab060ed13c147f74cefdb72548a12
I20260319 16:37:51.448896 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.445608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:44041 (local address 127.24.187.254:33813)
0319 16:37:51.445667 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.445669 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.445882 (+ 213us) server_negotiation.cc:408] Connection header received
0319 16:37:51.446350 (+ 468us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.446352 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.446361 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.446422 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:51.446837 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.447314 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.447915 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.448077 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.448610 (+ 533us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:51.448614 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:51.448614 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:51.448618 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:51.448633 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:51.448636 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:51.448683 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.448778 (+ 95us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.448814 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260319 16:37:51.449062 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.445571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:33813 (local address 127.24.187.195:44041)
0319 16:37:51.445789 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.445816 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.446274 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.446463 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.446466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.446474 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:51.446774 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.446777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.447405 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.447406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.447838 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.447840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.447938 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.448464 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:51.448465 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:51.448494 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:51.448512 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:51.448697 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:51.448702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:51.448705 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.448779 (+ 74us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.448845 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260319 16:37:51.449498 13341 heartbeater.cc:344] Connected to a master server at 127.24.187.254:33813
I20260319 16:37:51.449574 13341 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:51.449779 13341 heartbeater.cc:507] Master 127.24.187.254:33813 requested a full tablet report, sending...
I20260319 16:37:51.450168 12891 ts_manager.cc:194] Registered new tserver with Master: 30aea4ee124446718e27a95d1e2345ff (127.24.187.195:35357)
I20260319 16:37:51.450538 12891 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.195:44041
I20260319 16:37:51.460105 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:37:51.463771 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.461446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.24.187.254:33813)
0319 16:37:51.461503 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.461505 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.461559 (+ 54us) server_negotiation.cc:408] Connection header received
0319 16:37:51.461611 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.461612 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.461621 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.461659 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:51.461976 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.462464 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.462970 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.463076 (+ 106us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.463522 (+ 446us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:51.463525 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:51.463526 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:51.463530 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:51.463544 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:51.463547 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:51.463628 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.463694 (+ 66us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.463712 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260319 16:37:51.465080 12891 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:37:51.466331 12891 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:37:51.471132 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.468195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:36907 (local address 127.0.0.1:37054)
0319 16:37:51.468413 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.468532 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.468607 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.468856 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.468857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.468860 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:51.469074 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.469077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.470313 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.470314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.470869 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.470871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.470934 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.470966 (+ 32us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.471019 (+ 53us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.471064 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
I20260319 16:37:51.471908 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.468276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.24.187.193:36907)
0319 16:37:51.468733 (+ 457us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.468739 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.468746 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:37:51.468774 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.468776 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.468787 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.468839 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:51.469141 (+ 302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.470235 (+ 1094us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.470944 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.471734 (+ 790us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.471793 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.471817 (+ 24us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.471839 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":101,"threads_started":1}
I20260319 16:37:51.472108 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.468447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:38105 (local address 127.0.0.1:38468)
0319 16:37:51.468566 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.468575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.468622 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.468854 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.468856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.468859 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:51.469220 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.469222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.471013 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.471015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.471930 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.471931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.471993 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.471995 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.472023 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.472049 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260319 16:37:51.472116 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.468232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:35357 (local address 127.0.0.1:35692)
0319 16:37:51.468561 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.468569 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.468633 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.469125 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.469126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.469128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:51.469302 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.469304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.471011 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.471012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.471930 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.471931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.471974 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.471977 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.472000 (+ 23us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.472032 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":94,"threads_started":1}
I20260319 16:37:51.472406 13013 tablet_service.cc:1511] Processing CreateTablet for tablet b779b305559649fd8d76bbe62665225a (DEFAULT_TABLE table=test-table [id=dd72a305683646349848d6ec6e0e1dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:51.472671 13013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b779b305559649fd8d76bbe62665225a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:51.472877 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.468403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.24.187.195:35357)
0319 16:37:51.468977 (+ 574us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.468984 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.468993 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:51.469024 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.469026 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.469039 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.469375 (+ 336us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:51.469459 (+ 84us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.470896 (+ 1437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.472181 (+ 1285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.472665 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.472745 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.472773 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.472805 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":90,"threads_started":1}
I20260319 16:37:51.472965 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.468563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.24.187.194:38105)
0319 16:37:51.468725 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.468731 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.468738 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:37:51.468766 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.468768 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.468779 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.468833 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:51.469305 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.470936 (+ 1631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.472316 (+ 1380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.472787 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.472846 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.472869 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.472901 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260319 16:37:51.473642 13144 tablet_service.cc:1511] Processing CreateTablet for tablet b779b305559649fd8d76bbe62665225a (DEFAULT_TABLE table=test-table [id=dd72a305683646349848d6ec6e0e1dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:51.473670 13275 tablet_service.cc:1511] Processing CreateTablet for tablet b779b305559649fd8d76bbe62665225a (DEFAULT_TABLE table=test-table [id=dd72a305683646349848d6ec6e0e1dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:51.473919 13144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b779b305559649fd8d76bbe62665225a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:51.473945 13275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b779b305559649fd8d76bbe62665225a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:51.475472 13355 tablet_bootstrap.cc:492] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86: Bootstrap starting.
I20260319 16:37:51.476140 13355 tablet_bootstrap.cc:654] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:51.476342 13356 tablet_bootstrap.cc:492] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Bootstrap starting.
I20260319 16:37:51.476732 13355 log.cc:826] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:51.476768 13357 tablet_bootstrap.cc:492] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Bootstrap starting.
I20260319 16:37:51.476934 13356 tablet_bootstrap.cc:654] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:51.477213 13356 log.cc:826] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:51.477290 13357 tablet_bootstrap.cc:654] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:51.477371 13355 tablet_bootstrap.cc:492] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86: No bootstrap required, opened a new log
I20260319 16:37:51.477437 13355 ts_tablet_manager.cc:1403] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:51.477560 13357 log.cc:826] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:51.477929 13356 tablet_bootstrap.cc:492] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: No bootstrap required, opened a new log
I20260319 16:37:51.477994 13356 ts_tablet_manager.cc:1403] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:51.478289 13357 tablet_bootstrap.cc:492] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: No bootstrap required, opened a new log
I20260319 16:37:51.478341 13357 ts_tablet_manager.cc:1403] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:51.478963 13355 raft_consensus.cc:359] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.479107 13355 raft_consensus.cc:385] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:51.479135 13355 raft_consensus.cc:740] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d7282f85db431e922707c8215f0b86, State: Initialized, Role: FOLLOWER
I20260319 16:37:51.479254 13355 consensus_queue.cc:260] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.479415 13356 raft_consensus.cc:359] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.479497 13355 ts_tablet_manager.cc:1434] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:51.479539 13356 raft_consensus.cc:385] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:51.479564 13356 raft_consensus.cc:740] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695d99feb7a0418485a330213f5c9d9e, State: Initialized, Role: FOLLOWER
I20260319 16:37:51.479612 13079 heartbeater.cc:499] Master 127.24.187.254:33813 was elected leader, sending a full tablet report...
I20260319 16:37:51.479662 13356 consensus_queue.cc:260] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.479481 13357 raft_consensus.cc:359] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.479852 13357 raft_consensus.cc:385] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:51.479897 13357 raft_consensus.cc:740] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30aea4ee124446718e27a95d1e2345ff, State: Initialized, Role: FOLLOWER
I20260319 16:37:51.480038 13356 ts_tablet_manager.cc:1434] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:51.480075 13357 consensus_queue.cc:260] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.480136 13210 heartbeater.cc:499] Master 127.24.187.254:33813 was elected leader, sending a full tablet report...
I20260319 16:37:51.480309 13357 ts_tablet_manager.cc:1434] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:51.480406 13341 heartbeater.cc:499] Master 127.24.187.254:33813 was elected leader, sending a full tablet report...
W20260319 16:37:51.582528 13211 tablet.cc:2379] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:51.684849 13361 raft_consensus.cc:493] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:51.684976 13361 raft_consensus.cc:515] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.685321 13361 leader_election.cc:290] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 695d99feb7a0418485a330213f5c9d9e (127.24.187.194:38105), 30aea4ee124446718e27a95d1e2345ff (127.24.187.195:35357)
I20260319 16:37:51.687464 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.685533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:38105 (local address 127.24.187.193:33181)
0319 16:37:51.685596 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.685606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.685654 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.685841 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.685842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.685844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:51.686025 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.686027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.686708 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.686709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.687260 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.687261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.687330 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.687334 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.687361 (+ 27us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.687393 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260319 16:37:51.687824 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.685583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:33181 (local address 127.24.187.194:38105)
0319 16:37:51.685690 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.685692 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.685701 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:51.685752 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.685754 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.685768 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.685815 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:51.686094 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.686625 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.687338 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.687730 (+ 392us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.687739 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.687758 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.687779 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260319 16:37:51.688009 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.685551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:35357 (local address 127.24.187.193:38173)
0319 16:37:51.685685 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.685695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:51.685800 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:51.685995 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:51.685995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:51.686010 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:51.686188 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.686189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.686881 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.686881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:51.687801 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:51.687803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.687866 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.687870 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:51.687898 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:37:51.687932 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260319 16:37:51.688252 13164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b779b305559649fd8d76bbe62665225a" candidate_uuid: "73d7282f85db431e922707c8215f0b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695d99feb7a0418485a330213f5c9d9e" is_pre_election: true
I20260319 16:37:51.688400 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.685627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:38173 (local address 127.24.187.195:35357)
0319 16:37:51.685669 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.685672 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.685797 (+ 125us) server_negotiation.cc:408] Connection header received
0319 16:37:51.685897 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.685899 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.685912 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.685938 (+ 26us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:51.686262 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.686785 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.687882 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.688290 (+ 408us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.688297 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.688316 (+ 19us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.688338 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:37:51.688412 13164 raft_consensus.cc:2468] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d7282f85db431e922707c8215f0b86 in term 0.
I20260319 16:37:51.688647 12966 leader_election.cc:304] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 695d99feb7a0418485a330213f5c9d9e, 73d7282f85db431e922707c8215f0b86; no voters:
I20260319 16:37:51.688642 13295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b779b305559649fd8d76bbe62665225a" candidate_uuid: "73d7282f85db431e922707c8215f0b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30aea4ee124446718e27a95d1e2345ff" is_pre_election: true
I20260319 16:37:51.688755 13295 raft_consensus.cc:2468] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d7282f85db431e922707c8215f0b86 in term 0.
I20260319 16:37:51.688764 13361 raft_consensus.cc:2804] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:51.688813 13361 raft_consensus.cc:493] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:51.688853 13361 raft_consensus.cc:3060] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:51.689528 13361 raft_consensus.cc:515] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.689643 13361 leader_election.cc:290] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [CANDIDATE]: Term 1 election: Requested vote from peers 695d99feb7a0418485a330213f5c9d9e (127.24.187.194:38105), 30aea4ee124446718e27a95d1e2345ff (127.24.187.195:35357)
I20260319 16:37:51.689828 13295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b779b305559649fd8d76bbe62665225a" candidate_uuid: "73d7282f85db431e922707c8215f0b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30aea4ee124446718e27a95d1e2345ff"
I20260319 16:37:51.689893 13295 raft_consensus.cc:3060] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:51.690817 13295 raft_consensus.cc:2468] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d7282f85db431e922707c8215f0b86 in term 1.
I20260319 16:37:51.690932 13164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b779b305559649fd8d76bbe62665225a" candidate_uuid: "73d7282f85db431e922707c8215f0b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695d99feb7a0418485a330213f5c9d9e"
I20260319 16:37:51.691004 12967 leader_election.cc:304] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30aea4ee124446718e27a95d1e2345ff, 73d7282f85db431e922707c8215f0b86; no voters:
I20260319 16:37:51.691011 13164 raft_consensus.cc:3060] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:51.691118 13361 raft_consensus.cc:2804] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:51.691258 13361 raft_consensus.cc:697] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 1 LEADER]: Becoming Leader. State: Replica: 73d7282f85db431e922707c8215f0b86, State: Running, Role: LEADER
I20260319 16:37:51.691370 13361 consensus_queue.cc:237] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } }
I20260319 16:37:51.691805 13164 raft_consensus.cc:2468] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d7282f85db431e922707c8215f0b86 in term 1.
I20260319 16:37:51.692078 12890 catalog_manager.cc:5671] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d7282f85db431e922707c8215f0b86 (127.24.187.193). New cstate: current_term: 1 leader_uuid: "73d7282f85db431e922707c8215f0b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7282f85db431e922707c8215f0b86" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 36907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 } health_report { overall_health: UNKNOWN } } }
W20260319 16:37:51.695393 13342 tablet.cc:2379] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:51.700030 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.697990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.24.187.193:36907)
0319 16:37:51.698026 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:51.698028 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:51.698038 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:37:51.698072 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:51.698073 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:51.698089 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:51.698137 (+ 48us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:51.698326 (+ 189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.698828 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:51.699446 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:51.699583 (+ 137us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:51.699734 (+ 151us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:51.699812 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:37:51.699871 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:51.699942 (+ 71us) server_negotiation.cc:300] Negotiation successful
0319 16:37:51.699959 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:37:51.701540 13295 raft_consensus.cc:1275] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 1 FOLLOWER]: Refusing update from remote peer 73d7282f85db431e922707c8215f0b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:51.701550 13164 raft_consensus.cc:1275] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 1 FOLLOWER]: Refusing update from remote peer 73d7282f85db431e922707c8215f0b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:51.701854 13365 consensus_queue.cc:1048] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "695d99feb7a0418485a330213f5c9d9e" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:51.701972 13361 consensus_queue.cc:1048] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30aea4ee124446718e27a95d1e2345ff" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:51.704593 13370 mvcc.cc:204] Tried to move back new op lower bound from 7266051160887091200 to 7266051160848498688. Current Snapshot: MvccSnapshot[applied={T|T < 7266051160887091200}]
I20260319 16:37:51.705152 13369 mvcc.cc:204] Tried to move back new op lower bound from 7266051160887091200 to 7266051160848498688. Current Snapshot: MvccSnapshot[applied={T|T < 7266051160887091200}]
I20260319 16:37:51.706010 12890 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:51.706089 12890 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:51.706739 12890 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6e7921859f284cc5bf1cc2ebc7130ba1: Sending DeleteTablet for 3 replicas of tablet b779b305559649fd8d76bbe62665225a
I20260319 16:37:51.707073 13144 tablet_service.cc:1558] Processing DeleteTablet for tablet b779b305559649fd8d76bbe62665225a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:51 UTC) from {username='slave'} at 127.0.0.1:38468
I20260319 16:37:51.707079 13275 tablet_service.cc:1558] Processing DeleteTablet for tablet b779b305559649fd8d76bbe62665225a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:51 UTC) from {username='slave'} at 127.0.0.1:35692
I20260319 16:37:51.707113 13013 tablet_service.cc:1558] Processing DeleteTablet for tablet b779b305559649fd8d76bbe62665225a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:51 UTC) from {username='slave'} at 127.0.0.1:37054
I20260319 16:37:51.707288 13378 tablet_replica.cc:333] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: stopping tablet replica
I20260319 16:37:51.707355 13378 raft_consensus.cc:2243] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:51.707444 13378 raft_consensus.cc:2272] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:51.707434 13379 tablet_replica.cc:333] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86: stopping tablet replica
I20260319 16:37:51.707564 13379 raft_consensus.cc:2243] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:37:51.707685 13379 raft_consensus.cc:2272] T b779b305559649fd8d76bbe62665225a P 73d7282f85db431e922707c8215f0b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:51.707751 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 12951
I20260319 16:37:51.707842 13378 ts_tablet_manager.cc:1916] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:51.708253 13380 tablet_replica.cc:333] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: stopping tablet replica
I20260319 16:37:51.708338 13380 raft_consensus.cc:2243] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:51.708710 13380 raft_consensus.cc:2272] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:51.709081 13380 ts_tablet_manager.cc:1916] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:51.709326 13378 ts_tablet_manager.cc:1929] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:51.709383 13378 log.cc:1199] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/b779b305559649fd8d76bbe62665225a
I20260319 16:37:51.709589 13378 ts_tablet_manager.cc:1950] T b779b305559649fd8d76bbe62665225a P 695d99feb7a0418485a330213f5c9d9e: Deleting consensus metadata
I20260319 16:37:51.710469 13380 ts_tablet_manager.cc:1929] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:51.710533 13380 log.cc:1199] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/b779b305559649fd8d76bbe62665225a
I20260319 16:37:51.710772 13380 ts_tablet_manager.cc:1950] T b779b305559649fd8d76bbe62665225a P 30aea4ee124446718e27a95d1e2345ff: Deleting consensus metadata
I20260319 16:37:51.711081 12875 catalog_manager.cc:5002] TS 695d99feb7a0418485a330213f5c9d9e (127.24.187.194:38105): tablet b779b305559649fd8d76bbe62665225a (table test-table [id=dd72a305683646349848d6ec6e0e1dfc]) successfully deleted
I20260319 16:37:51.711135 12876 catalog_manager.cc:5002] TS 30aea4ee124446718e27a95d1e2345ff (127.24.187.195:35357): tablet b779b305559649fd8d76bbe62665225a (table test-table [id=dd72a305683646349848d6ec6e0e1dfc]) successfully deleted
W20260319 16:37:51.713865 12875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:36907 (error 108)
I20260319 16:37:51.713969 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 13082
I20260319 16:37:51.715078 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:51.714933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:36907 (local address 127.0.0.1:37082)
0319 16:37:51.714992 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:51.715042 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:36907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260319 16:37:51.715215 12875 catalog_manager.cc:4729] TS 73d7282f85db431e922707c8215f0b86 (127.24.187.193:36907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b779b305559649fd8d76bbe62665225a: Network error: Client connection negotiation failed: client connection to 127.24.187.193:36907: connect: Connection refused (error 111)
I20260319 16:37:51.718899 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 13213
I20260319 16:37:51.722966 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 12860
2026-03-19T16:37:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (819 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:37:51 dist-test-slave-svn8 krb5kdc[13384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:37:51 dist-test-slave-svn8 krb5kdc[13384](info): set up 2 sockets
Mar 19 16:37:51 dist-test-slave-svn8 krb5kdc[13384](info): commencing operation
krb5kdc: starting...
W20260319 16:37:53.578523 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.837s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:37:53 dist-test-slave-svn8 krb5kdc[13384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:37:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:53.708416 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:46121
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:53.782249 13400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:53.782428 13400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:53.782456 13400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:53.784005 13400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:53.784049 13400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:53.784067 13400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:53.784081 13400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:53.784096 13400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:53.785748 13400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:53.785955 13400 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:53.786105 13400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:53.788470 13408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:53.788429 13406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:53.788651 13400 server_base.cc:1060] running on GCE node
W20260319 16:37:53.788699 13405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:53.788902 13400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:53.789134 13400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:53.790272 13400 hybrid_clock.cc:648] HybridClock initialized: now 1773938273790248 us; error 37 us; skew 500 ppm
Mar 19 16:37:53 dist-test-slave-svn8 krb5kdc[13384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938273, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:53.792518 13400 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:37:53.792997 13400 webserver.cc:492] Webserver started at http://127.24.187.254:33383/ using document root <none> and password file <none>
I20260319 16:37:53.793150 13400 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:53.793179 13400 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:53.793259 13400 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:53.794152 13400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "80c46509f41947e5bec714e554e123ac"
format_stamp: "Formatted at 2026-03-19 16:37:53 on dist-test-slave-svn8"
server_key: "51cb5882ac5a566f19b3b1da06171c25"
server_key_iv: "e9f33a8d6b78e22e190ffc8ce8fc7072"
server_key_version: "encryptionkey@0"
I20260319 16:37:53.794417 13400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "80c46509f41947e5bec714e554e123ac"
format_stamp: "Formatted at 2026-03-19 16:37:53 on dist-test-slave-svn8"
server_key: "51cb5882ac5a566f19b3b1da06171c25"
server_key_iv: "e9f33a8d6b78e22e190ffc8ce8fc7072"
server_key_version: "encryptionkey@0"
I20260319 16:37:53.795507 13400 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:53.796167 13415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:53.796384 13400 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:53.796438 13400 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "80c46509f41947e5bec714e554e123ac"
format_stamp: "Formatted at 2026-03-19 16:37:53 on dist-test-slave-svn8"
server_key: "51cb5882ac5a566f19b3b1da06171c25"
server_key_iv: "e9f33a8d6b78e22e190ffc8ce8fc7072"
server_key_version: "encryptionkey@0"
I20260319 16:37:53.796486 13400 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:53.818512 13400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:53.820310 13400 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:53.820453 13400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:53.824038 13400 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:46121
I20260319 16:37:53.824074 13467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:46121 every 8 connection(s)
I20260319 16:37:53.824360 13400 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:53.824899 13468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:53.827246 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac: Bootstrap starting.
I20260319 16:37:53.827790 13468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:53.828020 13468 log.cc:826] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:53.828670 13468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac: No bootstrap required, opened a new log
I20260319 16:37:53.829867 13468 raft_consensus.cc:359] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c46509f41947e5bec714e554e123ac" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46121 } }
I20260319 16:37:53.829977 13468 raft_consensus.cc:385] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:53.830004 13468 raft_consensus.cc:740] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c46509f41947e5bec714e554e123ac, State: Initialized, Role: FOLLOWER
I20260319 16:37:53.830102 13468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c46509f41947e5bec714e554e123ac" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46121 } }
I20260319 16:37:53.830171 13468 raft_consensus.cc:399] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:53.830199 13468 raft_consensus.cc:493] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:53.830241 13468 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:53.830771 13468 raft_consensus.cc:515] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c46509f41947e5bec714e554e123ac" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46121 } }
I20260319 16:37:53.830875 13468 leader_election.cc:304] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80c46509f41947e5bec714e554e123ac; no voters:
I20260319 16:37:53.831001 13468 leader_election.cc:290] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:53.831072 13471 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:53.831216 13468 sys_catalog.cc:565] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:53.831219 13471 raft_consensus.cc:697] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [term 1 LEADER]: Becoming Leader. State: Replica: 80c46509f41947e5bec714e554e123ac, State: Running, Role: LEADER
I20260319 16:37:53.831416 13471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c46509f41947e5bec714e554e123ac" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46121 } }
I20260319 16:37:53.831764 13472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80c46509f41947e5bec714e554e123ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c46509f41947e5bec714e554e123ac" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46121 } } }
I20260319 16:37:53.831844 13472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:53.831990 13473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80c46509f41947e5bec714e554e123ac. Latest consensus state: current_term: 1 leader_uuid: "80c46509f41947e5bec714e554e123ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c46509f41947e5bec714e554e123ac" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46121 } } }
I20260319 16:37:53.832015 13479 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:53.832044 13473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:53.832491 13479 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:53.832630 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 13400
I20260319 16:37:53.832765 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:53.832944 25327 external_mini_cluster.cc:1468] Setting key 7be172a886707c4533999bf02c3d360f
I20260319 16:37:53.833952 13479 catalog_manager.cc:1357] Generated new cluster ID: c252c92d5f83492f9923bfc37464e9a0
I20260319 16:37:53.834002 13479 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:37:53 dist-test-slave-svn8 krb5kdc[13384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:53.840904 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:53.833411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.24.187.254:46121)
0319 16:37:53.833564 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:53.833575 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:53.833763 (+ 188us) server_negotiation.cc:408] Connection header received
0319 16:37:53.833900 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:53.833906 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:53.833954 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:53.834058 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:53.834824 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:53.835566 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:53.836091 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:53.836567 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:53.838263 (+ 1696us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:53.838286 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:53.838287 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:53.838301 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:53.839775 (+ 1474us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:53.840066 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:53.840069 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:53.840069 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:53.840114 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:53.840251 (+ 137us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:53.840252 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:53.840252 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:53.840416 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:53.840557 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:53.840633 (+ 76us) server_negotiation.cc:300] Negotiation successful
0319 16:37:53.840747 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260319 16:37:53.855149 13479 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:53.855686 13479 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:53.861866 13479 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac: Generated new TSK 0
I20260319 16:37:53.862025 13479 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:53.918392 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:46121
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:54.014787 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:54.014984 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:54.015017 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:54.016525 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:54.016578 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:54.016615 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:54.018326 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:54.018556 13496 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:54.018724 13496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:54.021139 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.021207 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.021302 13496 server_base.cc:1060] running on GCE node
W20260319 16:37:54.021195 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.021529 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:54.021700 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:54.022876 13496 hybrid_clock.cc:648] HybridClock initialized: now 1773938274022862 us; error 26 us; skew 500 ppm
Mar 19 16:37:54 dist-test-slave-svn8 krb5kdc[13384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938274, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:54.025497 13496 init.cc:377] Logged in from keytab as kudu/127.24.187.193@KRBTEST.COM (short username kudu)
I20260319 16:37:54.026017 13496 webserver.cc:492] Webserver started at http://127.24.187.193:41939/ using document root <none> and password file <none>
I20260319 16:37:54.026180 13496 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:54.026216 13496 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:54.026330 13496 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:54.027228 13496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "03251b19de35434da09091dbbffbd82f"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "5219a41d39b88070619ab80a525ed1f3"
server_key_iv: "f6f4c8314c26a29ab907aef17dfad6bd"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.027510 13496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "03251b19de35434da09091dbbffbd82f"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "5219a41d39b88070619ab80a525ed1f3"
server_key_iv: "f6f4c8314c26a29ab907aef17dfad6bd"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.028579 13496 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:37:54.029294 13511 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.029505 13496 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:54.029567 13496 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "03251b19de35434da09091dbbffbd82f"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "5219a41d39b88070619ab80a525ed1f3"
server_key_iv: "f6f4c8314c26a29ab907aef17dfad6bd"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.029625 13496 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:54.048573 13496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:54.050522 13496 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:54.050657 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:54.050871 13496 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:54.051182 13496 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:54.051215 13496 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.051244 13496 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:54.051261 13496 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.056710 13496 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:35837
I20260319 16:37:54.056779 13624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:35837 every 8 connection(s)
I20260319 16:37:54.057034 13496 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
Mar 19 16:37:54 dist-test-slave-svn8 krb5kdc[13384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938274, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.193@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:54.063239 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 13496
I20260319 16:37:54.063344 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:37:54.063545 25327 external_mini_cluster.cc:1468] Setting key 78338e371392aa5a4bb092207874fbd9
I20260319 16:37:54.066193 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.057777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:54431 (local address 127.24.187.254:46121)
0319 16:37:54.057856 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.057859 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.057972 (+ 113us) server_negotiation.cc:408] Connection header received
0319 16:37:54.058717 (+ 745us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.058719 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.058742 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.058802 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:54.059571 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.060069 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.060620 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.060799 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.063185 (+ 2386us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:54.063189 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:54.063189 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:54.063197 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:54.064801 (+ 1604us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.065229 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.065230 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.065230 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.065256 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.065501 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.065502 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.065502 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.065595 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:54.065646 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.065989 (+ 343us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.066094 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260319 16:37:54.066226 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.057681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46121 (local address 127.24.187.193:54431)
0319 16:37:54.057856 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.057890 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.058565 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.058853 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.058857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.059171 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:54.059461 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.059465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.060154 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.060155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.060538 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.060540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.060659 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.061178 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:54.061195 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:54.063076 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:54.064881 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:54.064888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:54.064890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:54.065143 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:54.065362 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:54.065363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:54.065363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:54.065442 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:54.065648 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:54.065653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:54.065828 (+ 175us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.065954 (+ 126us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.066068 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260319 16:37:54.067001 13625 heartbeater.cc:344] Connected to a master server at 127.24.187.254:46121
I20260319 16:37:54.067101 13625 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:54.067296 13625 heartbeater.cc:507] Master 127.24.187.254:46121 requested a full tablet report, sending...
I20260319 16:37:54.067847 13432 ts_manager.cc:194] Registered new tserver with Master: 03251b19de35434da09091dbbffbd82f (127.24.187.193:35837)
I20260319 16:37:54.068485 13432 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.193@KRBTEST.COM'} at 127.24.187.193:54431
WARNING: no policy specified for kudu/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:54.106350 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:46121
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:54.181214 13632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:54.181397 13632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:54.181422 13632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:54.182915 13632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:54.182964 13632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:54.182991 13632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:54.184685 13632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:54.184878 13632 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:54.185047 13632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:54.187574 13638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.187618 13637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.187666 13632 server_base.cc:1060] running on GCE node
W20260319 16:37:54.187551 13640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.187920 13632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:54.188090 13632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:54.189231 13632 hybrid_clock.cc:648] HybridClock initialized: now 1773938274189211 us; error 33 us; skew 500 ppm
Mar 19 16:37:54 dist-test-slave-svn8 krb5kdc[13384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938274, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:54.191458 13632 init.cc:377] Logged in from keytab as kudu/127.24.187.194@KRBTEST.COM (short username kudu)
I20260319 16:37:54.191917 13632 webserver.cc:492] Webserver started at http://127.24.187.194:41461/ using document root <none> and password file <none>
I20260319 16:37:54.192073 13632 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:54.192106 13632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:54.192191 13632 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:54.193080 13632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "c1db5d2a17804296ba09ed36c946121d"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "14835f066378f0d01aadaba6d0efbcf6"
server_key_iv: "590cd0143573e98a91d516bb2b95cd4e"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.193383 13632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "c1db5d2a17804296ba09ed36c946121d"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "14835f066378f0d01aadaba6d0efbcf6"
server_key_iv: "590cd0143573e98a91d516bb2b95cd4e"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.194504 13632 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:37:54.195206 13647 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.195405 13632 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:54.195467 13632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "c1db5d2a17804296ba09ed36c946121d"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "14835f066378f0d01aadaba6d0efbcf6"
server_key_iv: "590cd0143573e98a91d516bb2b95cd4e"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.195597 13632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:54.202382 13632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:54.204213 13632 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:54.204351 13632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:54.204548 13632 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:54.204835 13632 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:54.204866 13632 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.204893 13632 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:54.204916 13632 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.210470 13632 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:46665
I20260319 16:37:54.210541 13760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:46665 every 8 connection(s)
I20260319 16:37:54.210796 13632 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
Mar 19 16:37:54 dist-test-slave-svn8 krb5kdc[13384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938274, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.194@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:54.219085 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.211409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:35197 (local address 127.24.187.254:46121)
0319 16:37:54.211530 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.211533 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.211671 (+ 138us) server_negotiation.cc:408] Connection header received
0319 16:37:54.212181 (+ 510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.212182 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.212194 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.212245 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:54.212983 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.213493 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.214059 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.214219 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.216591 (+ 2372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:54.216595 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:54.216595 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:54.216603 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:54.217809 (+ 1206us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.218243 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.218244 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.218245 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.218270 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.218482 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.218483 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.218484 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.218587 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:54.218644 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.218899 (+ 255us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.218991 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260319 16:37:54.219203 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.211357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46121 (local address 127.24.187.194:35197)
0319 16:37:54.211572 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.211599 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.212091 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.212298 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.212303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.212612 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:54.212888 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.212892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.213576 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.213577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.213984 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.213986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.214102 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.214633 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:54.214649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:54.216481 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:54.217892 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:54.217897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:54.217899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:54.218152 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:54.218361 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:54.218362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:54.218362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:54.218426 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:54.218641 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:54.218646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:54.218759 (+ 113us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.218882 (+ 123us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.219015 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260319 16:37:54.219732 13761 heartbeater.cc:344] Connected to a master server at 127.24.187.254:46121
I20260319 16:37:54.219846 13761 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:54.220024 13761 heartbeater.cc:507] Master 127.24.187.254:46121 requested a full tablet report, sending...
I20260319 16:37:54.220425 13432 ts_manager.cc:194] Registered new tserver with Master: c1db5d2a17804296ba09ed36c946121d (127.24.187.194:46665)
I20260319 16:37:54.220726 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 13632
I20260319 16:37:54.220826 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:37:54.220916 13432 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.194@KRBTEST.COM'} at 127.24.187.194:35197
I20260319 16:37:54.221024 25327 external_mini_cluster.cc:1468] Setting key 3ea9752c4952dafa3087818cfac596dc
WARNING: no policy specified for kudu/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:54.266706 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:46121
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:54.342231 13768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:54.342414 13768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:54.342444 13768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:54.343914 13768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:54.343957 13768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:54.343991 13768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:54.345611 13768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.24.187.254:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:54.345847 13768 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:54.346010 13768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:54.348572 13776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.348613 13774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.348572 13773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.348686 13768 server_base.cc:1060] running on GCE node
I20260319 16:37:54.348937 13768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:54.349118 13768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:54.350272 13768 hybrid_clock.cc:648] HybridClock initialized: now 1773938274350253 us; error 24 us; skew 500 ppm
Mar 19 16:37:54 dist-test-slave-svn8 krb5kdc[13384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938274, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:54.352561 13768 init.cc:377] Logged in from keytab as kudu/127.24.187.195@KRBTEST.COM (short username kudu)
I20260319 16:37:54.353078 13768 webserver.cc:492] Webserver started at http://127.24.187.195:41233/ using document root <none> and password file <none>
I20260319 16:37:54.353255 13768 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:54.353291 13768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:54.353391 13768 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:54.354317 13768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "261145944c7741618717c60725565191"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "2ee3ad8ab5b16d59a87c538a42de4fb4"
server_key_iv: "02b8c282fb59c7f820972bf12f7b39b0"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.354617 13768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "261145944c7741618717c60725565191"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "2ee3ad8ab5b16d59a87c538a42de4fb4"
server_key_iv: "02b8c282fb59c7f820972bf12f7b39b0"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.355667 13768 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:54.356398 13783 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.356600 13768 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:54.356659 13768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "261145944c7741618717c60725565191"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "2ee3ad8ab5b16d59a87c538a42de4fb4"
server_key_iv: "02b8c282fb59c7f820972bf12f7b39b0"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.356720 13768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:54.369699 13768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:54.371295 13768 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:54.371414 13768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:54.371620 13768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:54.371909 13768 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:54.371940 13768 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.371968 13768 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:54.371990 13768 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.377477 13768 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:35917
I20260319 16:37:54.377537 13896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:35917 every 8 connection(s)
I20260319 16:37:54.377833 13768 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:54.381312 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 13768
I20260319 16:37:54.381419 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:37:54.381578 25327 external_mini_cluster.cc:1468] Setting key 04c987a09f9b4773825679a068f4659e
Mar 19 16:37:54 dist-test-slave-svn8 krb5kdc[13384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938274, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.195@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:37:54.386024 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.378462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:37341 (local address 127.24.187.254:46121)
0319 16:37:54.378525 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.378527 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.378651 (+ 124us) server_negotiation.cc:408] Connection header received
0319 16:37:54.379155 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.379157 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.379169 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.379223 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:54.379958 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.380433 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.380938 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.381098 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.383502 (+ 2404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:54.383505 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:54.383505 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:54.383512 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:54.384906 (+ 1394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.385250 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.385251 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.385252 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.385274 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.385430 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.385431 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.385432 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.385523 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:54.385578 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.385846 (+ 268us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.385931 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:37:54.386119 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.378375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:46121 (local address 127.24.187.195:37341)
0319 16:37:54.378545 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.378579 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.379070 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.379283 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.379286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.379601 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:54.379867 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.379870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.380508 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.380509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.380868 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.380870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.380972 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.381520 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:37:54.381535 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:54.383425 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:54.384987 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:54.384994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:54.384996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:54.385185 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:54.385324 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:37:54.385324 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:37:54.385325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:37:54.385381 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:37:54.385581 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:54.385585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:54.385690 (+ 105us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.385806 (+ 116us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.385933 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260319 16:37:54.386559 13897 heartbeater.cc:344] Connected to a master server at 127.24.187.254:46121
I20260319 16:37:54.386643 13897 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:54.386816 13897 heartbeater.cc:507] Master 127.24.187.254:46121 requested a full tablet report, sending...
I20260319 16:37:54.387192 13432 ts_manager.cc:194] Registered new tserver with Master: 261145944c7741618717c60725565191 (127.24.187.195:35917)
I20260319 16:37:54.387588 13432 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.187.195@KRBTEST.COM'} at 127.24.187.195:37341
I20260319 16:37:54.393244 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:37:54.399052 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.394492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.24.187.254:46121)
0319 16:37:54.394540 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.394542 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.394626 (+ 84us) server_negotiation.cc:408] Connection header received
0319 16:37:54.394666 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.394667 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.394675 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.394713 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:54.395284 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.395805 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.396327 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.396439 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.397252 (+ 813us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:54.397254 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:54.397254 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:54.397259 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:54.398312 (+ 1053us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.398580 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.398581 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.398581 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.398600 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:54.398692 (+ 92us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:54.398692 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:54.398692 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:54.398789 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:54.398860 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.398931 (+ 71us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.398992 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260319 16:37:54.400506 13432 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:37:54.401734 13432 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:37:54.406916 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.403668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:35917 (local address 127.0.0.1:47502)
0319 16:37:54.404021 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.404042 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.404185 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.404476 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.404478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.404480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:54.404694 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.404697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.405560 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.405561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.406353 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.406355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.406428 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.406744 (+ 316us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.406794 (+ 50us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.406836 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":94,"threads_started":1}
I20260319 16:37:54.407153 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.403884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.24.187.195:35917)
0319 16:37:54.404333 (+ 449us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.404339 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.404348 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:54.404382 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.404383 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.404397 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.404456 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:54.404765 (+ 309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.405484 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.406456 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.406964 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.407014 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.407041 (+ 27us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.407082 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":74,"threads_started":1}
I20260319 16:37:54.407341 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.403668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:35837 (local address 127.0.0.1:42450)
0319 16:37:54.404011 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.404040 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.404115 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.404310 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.404312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.404321 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:54.404588 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.404590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.406036 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.406037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.406752 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.406755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.406820 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.406825 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.406855 (+ 30us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.406894 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":67,"threads_started":1}
I20260319 16:37:54.407637 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.403836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.24.187.193:35837)
0319 16:37:54.404017 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.404024 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.404144 (+ 120us) server_negotiation.cc:408] Connection header received
0319 16:37:54.404206 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.404208 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.404225 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.404301 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:54.404675 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.405968 (+ 1293us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.406973 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.407432 (+ 459us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.407480 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.407509 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.407551 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260319 16:37:54.407956 13831 tablet_service.cc:1511] Processing CreateTablet for tablet d7bf56ff5bf74344963af058706e9ccd (DEFAULT_TABLE table=test-table [id=aa60d9b39c6c4989bada1806aed4c1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:54.408073 13559 tablet_service.cc:1511] Processing CreateTablet for tablet d7bf56ff5bf74344963af058706e9ccd (DEFAULT_TABLE table=test-table [id=aa60d9b39c6c4989bada1806aed4c1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:54.408245 13831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7bf56ff5bf74344963af058706e9ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:54.408306 13559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7bf56ff5bf74344963af058706e9ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:54.411036 13915 tablet_bootstrap.cc:492] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Bootstrap starting.
I20260319 16:37:54.411154 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.403683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:46665 (local address 127.0.0.1:45444)
0319 16:37:54.404169 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.404176 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.404217 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.404427 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.404427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.404429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:54.404617 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.404618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.405948 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.405949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.406763 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.406766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.406823 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.406826 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.406852 (+ 26us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.406883 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":83,"threads_started":1}
I20260319 16:37:54.411301 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.403899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.24.187.194:46665)
0319 16:37:54.404047 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.404054 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.404229 (+ 175us) server_negotiation.cc:408] Connection header received
0319 16:37:54.404269 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.404271 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.404287 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.404348 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:54.404689 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.405872 (+ 1183us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.406958 (+ 1086us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.407450 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.407484 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.407512 (+ 28us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.407565 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260319 16:37:54.411655 13915 tablet_bootstrap.cc:654] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:54.411775 13695 tablet_service.cc:1511] Processing CreateTablet for tablet d7bf56ff5bf74344963af058706e9ccd (DEFAULT_TABLE table=test-table [id=aa60d9b39c6c4989bada1806aed4c1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:54.411895 13915 log.cc:826] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:54.412005 13695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7bf56ff5bf74344963af058706e9ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:54.412173 13914 tablet_bootstrap.cc:492] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f: Bootstrap starting.
I20260319 16:37:54.412722 13914 tablet_bootstrap.cc:654] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:54.412727 13915 tablet_bootstrap.cc:492] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: No bootstrap required, opened a new log
I20260319 16:37:54.412842 13915 ts_tablet_manager.cc:1403] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:54.413091 13914 log.cc:826] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:54.413810 13914 tablet_bootstrap.cc:492] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f: No bootstrap required, opened a new log
I20260319 16:37:54.413872 13914 ts_tablet_manager.cc:1403] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:54.414361 13915 raft_consensus.cc:359] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.414502 13915 raft_consensus.cc:385] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:54.414536 13915 raft_consensus.cc:740] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261145944c7741618717c60725565191, State: Initialized, Role: FOLLOWER
I20260319 16:37:54.414470 13918 tablet_bootstrap.cc:492] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Bootstrap starting.
I20260319 16:37:54.414624 13915 consensus_queue.cc:260] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.414856 13915 ts_tablet_manager.cc:1434] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:54.414887 13897 heartbeater.cc:499] Master 127.24.187.254:46121 was elected leader, sending a full tablet report...
I20260319 16:37:54.415158 13918 tablet_bootstrap.cc:654] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:54.415406 13918 log.cc:826] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:54.415530 13914 raft_consensus.cc:359] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.415666 13914 raft_consensus.cc:385] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:54.415704 13914 raft_consensus.cc:740] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03251b19de35434da09091dbbffbd82f, State: Initialized, Role: FOLLOWER
I20260319 16:37:54.415807 13914 consensus_queue.cc:260] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.416035 13914 ts_tablet_manager.cc:1434] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:54.416059 13918 tablet_bootstrap.cc:492] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: No bootstrap required, opened a new log
I20260319 16:37:54.416105 13918 ts_tablet_manager.cc:1403] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:54.416167 13625 heartbeater.cc:499] Master 127.24.187.254:46121 was elected leader, sending a full tablet report...
I20260319 16:37:54.417167 13918 raft_consensus.cc:359] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.417268 13918 raft_consensus.cc:385] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:54.417286 13918 raft_consensus.cc:740] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1db5d2a17804296ba09ed36c946121d, State: Initialized, Role: FOLLOWER
I20260319 16:37:54.417359 13918 consensus_queue.cc:260] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.417551 13918 ts_tablet_manager.cc:1434] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:54.417696 13761 heartbeater.cc:499] Master 127.24.187.254:46121 was elected leader, sending a full tablet report...
W20260319 16:37:54.461474 13762 tablet.cc:2379] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:54.512768 13922 raft_consensus.cc:493] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:54.512907 13922 raft_consensus.cc:515] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.513410 13922 leader_election.cc:290] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 261145944c7741618717c60725565191 (127.24.187.195:35917), 03251b19de35434da09091dbbffbd82f (127.24.187.193:35837)
I20260319 16:37:54.516700 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.513616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:35917 (local address 127.24.187.194:39221)
0319 16:37:54.513871 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.513882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.514001 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.514129 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.514130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.514132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:54.514322 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.514323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.515498 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.515499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.516405 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.516408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.516524 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.516529 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.516563 (+ 34us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.516595 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260319 16:37:54.516709 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.513632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:35837 (local address 127.24.187.194:41753)
0319 16:37:54.513910 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.513921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.513986 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.514542 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.514543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.514545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:54.514768 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.514769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.515592 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.515593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.516469 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.516470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.516529 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.516531 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.516559 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.516595 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":139,"threads_started":1}
I20260319 16:37:54.517097 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.513757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:39221 (local address 127.24.187.195:35917)
0319 16:37:54.513802 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.513805 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.513980 (+ 175us) server_negotiation.cc:408] Connection header received
0319 16:37:54.514065 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.514067 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.514077 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.514125 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:54.514395 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.515403 (+ 1008us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.516493 (+ 1090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.516972 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.516980 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.517003 (+ 23us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.517033 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:37:54.517124 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.514144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:41753 (local address 127.24.187.193:35837)
0319 16:37:54.514425 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.514428 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.514436 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:37:54.514463 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.514465 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.514475 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.514522 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:54.514839 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.515512 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.516549 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.517019 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.517026 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.517044 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.517072 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260319 16:37:54.517452 13579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7bf56ff5bf74344963af058706e9ccd" candidate_uuid: "c1db5d2a17804296ba09ed36c946121d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03251b19de35434da09091dbbffbd82f" is_pre_election: true
I20260319 16:37:54.517531 13851 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7bf56ff5bf74344963af058706e9ccd" candidate_uuid: "c1db5d2a17804296ba09ed36c946121d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261145944c7741618717c60725565191" is_pre_election: true
I20260319 16:37:54.517618 13579 raft_consensus.cc:2468] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1db5d2a17804296ba09ed36c946121d in term 0.
I20260319 16:37:54.517645 13851 raft_consensus.cc:2468] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1db5d2a17804296ba09ed36c946121d in term 0.
I20260319 16:37:54.517854 13649 leader_election.cc:304] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03251b19de35434da09091dbbffbd82f, c1db5d2a17804296ba09ed36c946121d; no voters:
I20260319 16:37:54.518014 13922 raft_consensus.cc:2804] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:54.518054 13922 raft_consensus.cc:493] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:54.518075 13922 raft_consensus.cc:3060] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:54.518745 13922 raft_consensus.cc:515] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.518872 13922 leader_election.cc:290] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [CANDIDATE]: Term 1 election: Requested vote from peers 261145944c7741618717c60725565191 (127.24.187.195:35917), 03251b19de35434da09091dbbffbd82f (127.24.187.193:35837)
I20260319 16:37:54.519024 13851 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7bf56ff5bf74344963af058706e9ccd" candidate_uuid: "c1db5d2a17804296ba09ed36c946121d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261145944c7741618717c60725565191"
I20260319 16:37:54.519050 13579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7bf56ff5bf74344963af058706e9ccd" candidate_uuid: "c1db5d2a17804296ba09ed36c946121d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03251b19de35434da09091dbbffbd82f"
I20260319 16:37:54.519111 13579 raft_consensus.cc:3060] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:54.519111 13851 raft_consensus.cc:3060] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:54.519996 13851 raft_consensus.cc:2468] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1db5d2a17804296ba09ed36c946121d in term 1.
I20260319 16:37:54.519995 13579 raft_consensus.cc:2468] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1db5d2a17804296ba09ed36c946121d in term 1.
I20260319 16:37:54.520195 13651 leader_election.cc:304] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261145944c7741618717c60725565191, c1db5d2a17804296ba09ed36c946121d; no voters:
I20260319 16:37:54.520321 13922 raft_consensus.cc:2804] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:54.520499 13922 raft_consensus.cc:697] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 1 LEADER]: Becoming Leader. State: Replica: c1db5d2a17804296ba09ed36c946121d, State: Running, Role: LEADER
I20260319 16:37:54.520608 13922 consensus_queue.cc:237] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } }
I20260319 16:37:54.521443 13432 catalog_manager.cc:5671] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d reported cstate change: term changed from 0 to 1, leader changed from <none> to c1db5d2a17804296ba09ed36c946121d (127.24.187.194). New cstate: current_term: 1 leader_uuid: "c1db5d2a17804296ba09ed36c946121d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1db5d2a17804296ba09ed36c946121d" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 46665 } health_report { overall_health: HEALTHY } } }
I20260319 16:37:54.553233 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.551229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.24.187.194:46665)
0319 16:37:54.551310 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.551312 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.551322 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:37:54.551363 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.551364 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.551382 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.551439 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:54.551691 (+ 252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.552154 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.552750 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.552896 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.552937 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:54.553025 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:37:54.553098 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.553134 (+ 36us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.553157 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260319 16:37:54.554962 13579 raft_consensus.cc:1275] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 1 FOLLOWER]: Refusing update from remote peer c1db5d2a17804296ba09ed36c946121d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:54.554963 13851 raft_consensus.cc:1275] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 1 FOLLOWER]: Refusing update from remote peer c1db5d2a17804296ba09ed36c946121d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:54.555272 13922 consensus_queue.cc:1048] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [LEADER]: Connected to new peer: Peer: permanent_uuid: "261145944c7741618717c60725565191" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 35917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:54.555400 13924 consensus_queue.cc:1048] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03251b19de35434da09091dbbffbd82f" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:54.557855 13928 mvcc.cc:204] Tried to move back new op lower bound from 7266051172574646272 to 7266051172436901888. Current Snapshot: MvccSnapshot[applied={T|T < 7266051172574646272}]
I20260319 16:37:54.558504 13929 mvcc.cc:204] Tried to move back new op lower bound from 7266051172574646272 to 7266051172436901888. Current Snapshot: MvccSnapshot[applied={T|T < 7266051172574646272}]
I20260319 16:37:54.560364 13431 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:54.560446 13431 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:54.561184 13431 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 80c46509f41947e5bec714e554e123ac: Sending DeleteTablet for 3 replicas of tablet d7bf56ff5bf74344963af058706e9ccd
I20260319 16:37:54.561494 13831 tablet_service.cc:1558] Processing DeleteTablet for tablet d7bf56ff5bf74344963af058706e9ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:54 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:47502
I20260319 16:37:54.561640 13559 tablet_service.cc:1558] Processing DeleteTablet for tablet d7bf56ff5bf74344963af058706e9ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:54 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:42450
I20260319 16:37:54.561769 13695 tablet_service.cc:1558] Processing DeleteTablet for tablet d7bf56ff5bf74344963af058706e9ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:54 UTC) from {username='kudu', principal='kudu/127.24.187.254@KRBTEST.COM'} at 127.0.0.1:45444
I20260319 16:37:54.561967 13937 tablet_replica.cc:333] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f: stopping tablet replica
I20260319 16:37:54.561950 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 13496
I20260319 16:37:54.562044 13937 raft_consensus.cc:2243] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:54.562130 13937 raft_consensus.cc:2272] T d7bf56ff5bf74344963af058706e9ccd P 03251b19de35434da09091dbbffbd82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:54.563606 13938 tablet_replica.cc:333] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: stopping tablet replica
I20260319 16:37:54.563691 13938 raft_consensus.cc:2243] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:37:54.563796 13938 raft_consensus.cc:2272] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:54.564178 13938 ts_tablet_manager.cc:1916] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:54.565851 13938 ts_tablet_manager.cc:1929] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:54.565917 13938 log.cc:1199] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/d7bf56ff5bf74344963af058706e9ccd
I20260319 16:37:54.566161 13938 ts_tablet_manager.cc:1950] T d7bf56ff5bf74344963af058706e9ccd P c1db5d2a17804296ba09ed36c946121d: Deleting consensus metadata
I20260319 16:37:54.566511 13418 catalog_manager.cc:5002] TS c1db5d2a17804296ba09ed36c946121d (127.24.187.194:46665): tablet d7bf56ff5bf74344963af058706e9ccd (table test-table [id=aa60d9b39c6c4989bada1806aed4c1c9]) successfully deleted
W20260319 16:37:54.568287 13417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:35837 (error 108)
I20260319 16:37:54.568565 13936 tablet_replica.cc:333] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: stopping tablet replica
I20260319 16:37:54.568665 13936 raft_consensus.cc:2243] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:54.568773 13936 raft_consensus.cc:2272] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260319 16:37:54.568985 13908 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:37:54.568670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:35837 (local address 127.0.0.1:42456)
0319 16:37:54.568719 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.568730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.568767 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.568945 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:35837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
W20260319 16:37:54.569082 13417 catalog_manager.cc:4729] TS 03251b19de35434da09091dbbffbd82f (127.24.187.193:35837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7bf56ff5bf74344963af058706e9ccd: Network error: Client connection negotiation failed: client connection to 127.24.187.193:35837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260319 16:37:54.569340 13936 ts_tablet_manager.cc:1916] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:54.569602 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 13632
I20260319 16:37:54.570963 13936 ts_tablet_manager.cc:1929] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:54.571008 13936 log.cc:1199] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/d7bf56ff5bf74344963af058706e9ccd
I20260319 16:37:54.571190 13936 ts_tablet_manager.cc:1950] T d7bf56ff5bf74344963af058706e9ccd P 261145944c7741618717c60725565191: Deleting consensus metadata
I20260319 16:37:54.571722 13419 catalog_manager.cc:5002] TS 261145944c7741618717c60725565191 (127.24.187.195:35917): tablet d7bf56ff5bf74344963af058706e9ccd (table test-table [id=aa60d9b39c6c4989bada1806aed4c1c9]) successfully deleted
I20260319 16:37:54.574846 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 13768
I20260319 16:37:54.579560 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 13400
2026-03-19T16:37:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2856 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-03-19T16:37:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:54Z Disabled control of system clock
I20260319 16:37:54.604990 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42471
--webserver_interface=127.24.187.254
--webserver_port=0
--builtin_ntp_servers=127.24.187.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:54.677846 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:54.678002 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:54.678022 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:54.679462 13943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:54.679508 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:54.679520 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:54.679530 13943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:54.679541 13943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:54.681012 13943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:42471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:54.681177 13943 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:54.681341 13943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:54.683633 13948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.683657 13943 server_base.cc:1060] running on GCE node
W20260319 16:37:54.683681 13951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.683634 13949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.683969 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:54.684191 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:54.685354 13943 hybrid_clock.cc:648] HybridClock initialized: now 1773938274685332 us; error 36 us; skew 500 ppm
I20260319 16:37:54.686570 13943 webserver.cc:492] Webserver started at http://127.24.187.254:36657/ using document root <none> and password file <none>
I20260319 16:37:54.686764 13943 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:54.686805 13943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:54.686895 13943 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:54.687790 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "a856915d84c24eceb259553bbcf6c62e"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "617bbda43482223eea046a8eb47d61ed"
server_key_iv: "5329c1a13c7933f6452bfdf7feeab815"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.688060 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "a856915d84c24eceb259553bbcf6c62e"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "617bbda43482223eea046a8eb47d61ed"
server_key_iv: "5329c1a13c7933f6452bfdf7feeab815"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.689153 13943 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:37:54.689873 13957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.690011 13943 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:37:54.690069 13943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "a856915d84c24eceb259553bbcf6c62e"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "617bbda43482223eea046a8eb47d61ed"
server_key_iv: "5329c1a13c7933f6452bfdf7feeab815"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.690130 13943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:54.706049 13943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:54.706306 13943 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:54.706413 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:54.710140 13943 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:42471
I20260319 16:37:54.710170 14009 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:42471 every 8 connection(s)
I20260319 16:37:54.710461 13943 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:54.710976 14010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:54.713213 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e: Bootstrap starting.
I20260319 16:37:54.713768 14010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:54.713971 14010 log.cc:826] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:54.714547 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e: No bootstrap required, opened a new log
I20260319 16:37:54.715680 14010 raft_consensus.cc:359] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a856915d84c24eceb259553bbcf6c62e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42471 } }
I20260319 16:37:54.715793 14010 raft_consensus.cc:385] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:54.715812 14010 raft_consensus.cc:740] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a856915d84c24eceb259553bbcf6c62e, State: Initialized, Role: FOLLOWER
I20260319 16:37:54.715899 14010 consensus_queue.cc:260] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a856915d84c24eceb259553bbcf6c62e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42471 } }
I20260319 16:37:54.715961 14010 raft_consensus.cc:399] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:54.715988 14010 raft_consensus.cc:493] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:54.716017 14010 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:54.716471 14010 raft_consensus.cc:515] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a856915d84c24eceb259553bbcf6c62e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42471 } }
I20260319 16:37:54.716557 14010 leader_election.cc:304] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a856915d84c24eceb259553bbcf6c62e; no voters:
I20260319 16:37:54.716682 14010 leader_election.cc:290] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:54.716750 14013 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:54.716864 14010 sys_catalog.cc:565] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:54.716894 14013 raft_consensus.cc:697] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [term 1 LEADER]: Becoming Leader. State: Replica: a856915d84c24eceb259553bbcf6c62e, State: Running, Role: LEADER
I20260319 16:37:54.716967 14013 consensus_queue.cc:237] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a856915d84c24eceb259553bbcf6c62e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42471 } }
I20260319 16:37:54.717272 14015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a856915d84c24eceb259553bbcf6c62e. Latest consensus state: current_term: 1 leader_uuid: "a856915d84c24eceb259553bbcf6c62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a856915d84c24eceb259553bbcf6c62e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42471 } } }
I20260319 16:37:54.717286 14014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a856915d84c24eceb259553bbcf6c62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a856915d84c24eceb259553bbcf6c62e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 42471 } } }
I20260319 16:37:54.717360 14015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:54.717371 14014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:54.717918 14022 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:54.718353 14022 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:54.718745 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 13943
I20260319 16:37:54.718848 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:54.719036 25327 external_mini_cluster.cc:1468] Setting key 4b51978e1ea80814c02e40a49e574bc7
I20260319 16:37:54.719770 14022 catalog_manager.cc:1357] Generated new cluster ID: d3e0c5e085064c27a3145a21fd441e91
I20260319 16:37:54.719818 14022 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:37:54.725703 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.719562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.24.187.254:42471)
0319 16:37:54.719725 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.719732 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.721989 (+ 2257us) server_negotiation.cc:408] Connection header received
0319 16:37:54.722100 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.722106 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.722158 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.722285 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:54.722670 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.723830 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.724542 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.724799 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.725112 (+ 313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:54.725117 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:54.725118 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:54.725340 (+ 222us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:54.725374 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:54.725383 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:54.725457 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.725496 (+ 39us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.725548 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260319 16:37:54.744143 14022 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:54.744901 14022 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:54.754148 14022 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e: Generated new TSK 0
I20260319 16:37:54.754333 14022 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:37:54.760977 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.193:0
--local_ip_for_outbound_sockets=127.24.187.193
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42471
--builtin_ntp_servers=127.24.187.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260319 16:37:54.838410 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:54.838557 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:54.838577 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:54.839977 14034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:54.840015 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:54.840039 14034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.193
I20260319 16:37:54.841536 14034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.187.193
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.193
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:54.841727 14034 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:54.841909 14034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:54.844269 14042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.844259 14039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.844244 14040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.844473 14034 server_base.cc:1060] running on GCE node
I20260319 16:37:54.844664 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:54.844847 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:54.845999 14034 hybrid_clock.cc:648] HybridClock initialized: now 1773938274845981 us; error 32 us; skew 500 ppm
I20260319 16:37:54.847105 14034 webserver.cc:492] Webserver started at http://127.24.187.193:45357/ using document root <none> and password file <none>
I20260319 16:37:54.847285 14034 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:54.847325 14034 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:54.847425 14034 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:54.848316 14034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data/instance:
uuid: "2960aa2f3f3d461e8efd14266f910f87"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "bd334db7b587a6605d59bb504967ec8f"
server_key_iv: "2c4656cc9025cfedf29bcd50f2c775b9"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.848621 14034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance:
uuid: "2960aa2f3f3d461e8efd14266f910f87"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "bd334db7b587a6605d59bb504967ec8f"
server_key_iv: "2c4656cc9025cfedf29bcd50f2c775b9"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.849736 14034 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:54.850461 14048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.850639 14034 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:37:54.850696 14034 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/wal
uuid: "2960aa2f3f3d461e8efd14266f910f87"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "bd334db7b587a6605d59bb504967ec8f"
server_key_iv: "2c4656cc9025cfedf29bcd50f2c775b9"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.850754 14034 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:54.866923 14034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:54.867147 14034 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:54.867250 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:54.867430 14034 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:54.867729 14034 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:54.867759 14034 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.867787 14034 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:54.867805 14034 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.873090 14034 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.193:43149
I20260319 16:37:54.873157 14161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.193:43149 every 8 connection(s)
I20260319 16:37:54.873606 14034 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/data/info.pb
I20260319 16:37:54.875430 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14034
I20260319 16:37:54.875543 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-0/wal/instance
I20260319 16:37:54.875739 25327 external_mini_cluster.cc:1468] Setting key 9719679d9fad8c4a7773917a634dc6a5
I20260319 16:37:54.876655 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.194:0
--local_ip_for_outbound_sockets=127.24.187.194
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42471
--builtin_ntp_servers=127.24.187.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260319 16:37:54.877820 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.874422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.193:48105 (local address 127.24.187.254:42471)
0319 16:37:54.874477 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.874479 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.874527 (+ 48us) server_negotiation.cc:408] Connection header received
0319 16:37:54.875036 (+ 509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.875038 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.875052 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.875102 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:54.875576 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.876198 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.876874 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.877046 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.877551 (+ 505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:54.877554 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:54.877554 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:54.877559 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:54.877573 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:54.877577 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:54.877620 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.877695 (+ 75us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.877748 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:37:54.877935 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.874247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42471 (local address 127.24.187.193:48105)
0319 16:37:54.874423 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.874462 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.874937 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.875165 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.875168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.875176 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:54.875467 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.875469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.876290 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.876291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.876750 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.876754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.876908 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.877448 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:54.877448 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:54.877466 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:54.877482 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:54.877625 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:54.877629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:54.877633 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.877692 (+ 59us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.877791 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260319 16:37:54.878387 14162 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42471
I20260319 16:37:54.878476 14162 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:54.878657 14162 heartbeater.cc:507] Master 127.24.187.254:42471 requested a full tablet report, sending...
I20260319 16:37:54.879120 13974 ts_manager.cc:194] Registered new tserver with Master: 2960aa2f3f3d461e8efd14266f910f87 (127.24.187.193:43149)
I20260319 16:37:54.879719 13974 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.193:48105
W20260319 16:37:54.951084 14165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:54.951277 14165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:54.951304 14165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:54.953356 14165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:54.953401 14165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:54.953433 14165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.194
I20260319 16:37:54.955344 14165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.187.194
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.194
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:54.955564 14165 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:54.955816 14165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:54.958470 14170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.958494 14173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:54.958472 14171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:54.958768 14165 server_base.cc:1060] running on GCE node
I20260319 16:37:54.959050 14165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:54.959273 14165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:54.960374 14165 hybrid_clock.cc:648] HybridClock initialized: now 1773938274960366 us; error 31 us; skew 500 ppm
I20260319 16:37:54.961561 14165 webserver.cc:492] Webserver started at http://127.24.187.194:32899/ using document root <none> and password file <none>
I20260319 16:37:54.961807 14165 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:54.961855 14165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:54.961966 14165 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:54.963042 14165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data/instance:
uuid: "ebbeb4f96eb649c2bb61af6759af9041"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "93913574df5edfbc009d7c49f9ec49b9"
server_key_iv: "c99583ef1d1ac551e77966d38dc6bb57"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.963352 14165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance:
uuid: "ebbeb4f96eb649c2bb61af6759af9041"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "93913574df5edfbc009d7c49f9ec49b9"
server_key_iv: "c99583ef1d1ac551e77966d38dc6bb57"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.964723 14165 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:37:54.965526 14179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.965696 14165 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:37:54.965785 14165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal
uuid: "ebbeb4f96eb649c2bb61af6759af9041"
format_stamp: "Formatted at 2026-03-19 16:37:54 on dist-test-slave-svn8"
server_key: "93913574df5edfbc009d7c49f9ec49b9"
server_key_iv: "c99583ef1d1ac551e77966d38dc6bb57"
server_key_version: "encryptionkey@0"
I20260319 16:37:54.965844 14165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:54.981940 14165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:54.982149 14165 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:54.982266 14165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:54.982453 14165 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:54.982776 14165 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:54.982815 14165 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.982843 14165 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:54.982858 14165 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:54.989140 14165 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.194:44893
I20260319 16:37:54.989274 14292 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.194:44893 every 8 connection(s)
I20260319 16:37:54.989459 14165 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/data/info.pb
I20260319 16:37:54.991281 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14165
I20260319 16:37:54.991379 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal/instance
I20260319 16:37:54.991565 25327 external_mini_cluster.cc:1468] Setting key b9bb1f5ef574f5962ab75663d3c66393
I20260319 16:37:54.992434 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.187.195:0
--local_ip_for_outbound_sockets=127.24.187.195
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42471
--builtin_ntp_servers=127.24.187.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260319 16:37:54.993676 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.990115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:41679 (local address 127.24.187.254:42471)
0319 16:37:54.990167 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:54.990169 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:54.990400 (+ 231us) server_negotiation.cc:408] Connection header received
0319 16:37:54.990984 (+ 584us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:54.990985 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:54.991000 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:54.991052 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:54.991513 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.992113 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.992715 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.992864 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.993394 (+ 530us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:54.993398 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:54.993398 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:54.993403 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:54.993419 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:54.993423 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:54.993480 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:54.993547 (+ 67us) server_negotiation.cc:300] Negotiation successful
0319 16:37:54.993601 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:37:54.993798 14295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:54.990090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42471 (local address 127.24.187.194:41679)
0319 16:37:54.990267 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:54.990298 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:54.990905 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:54.991109 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:54.991113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:54.991120 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:54.991433 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.991437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.992192 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:54.992193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:54.992643 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:54.992645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:54.992733 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:54.993286 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:54.993286 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:54.993303 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:54.993320 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:54.993481 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:54.993483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:54.993485 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:37:54.993528 (+ 43us) client_negotiation.cc:240] Negotiation successful
0319 16:37:54.993603 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260319 16:37:54.994211 14293 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42471
I20260319 16:37:54.994298 14293 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:54.994481 14293 heartbeater.cc:507] Master 127.24.187.254:42471 requested a full tablet report, sending...
I20260319 16:37:54.994828 13974 ts_manager.cc:194] Registered new tserver with Master: ebbeb4f96eb649c2bb61af6759af9041 (127.24.187.194:44893)
I20260319 16:37:54.995275 13974 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.194:41679
W20260319 16:37:55.067589 14296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:55.067744 14296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:55.067763 14296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:55.069129 14296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:55.069165 14296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:55.069187 14296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.187.195
I20260319 16:37:55.070722 14296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.187.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.187.195
--webserver_port=0
--tserver_master_addrs=127.24.187.254:42471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.187.195
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:55.070886 14296 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:55.071056 14296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:55.073462 14301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:55.073532 14296 server_base.cc:1060] running on GCE node
W20260319 16:37:55.073673 14302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:55.073750 14304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:55.073925 14296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:55.074115 14296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:55.075335 14296 hybrid_clock.cc:648] HybridClock initialized: now 1773938275075293 us; error 35 us; skew 500 ppm
I20260319 16:37:55.076362 14296 webserver.cc:492] Webserver started at http://127.24.187.195:34513/ using document root <none> and password file <none>
I20260319 16:37:55.076532 14296 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:55.076571 14296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:55.076664 14296 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:55.077607 14296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data/instance:
uuid: "791286772f5e4f23852249e7d7bb8294"
format_stamp: "Formatted at 2026-03-19 16:37:55 on dist-test-slave-svn8"
server_key: "fd9ef89c37b8354268d9b3700e13a0fb"
server_key_iv: "9ab415785ab9f979e587e2fc8b1a2609"
server_key_version: "encryptionkey@0"
I20260319 16:37:55.077924 14296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance:
uuid: "791286772f5e4f23852249e7d7bb8294"
format_stamp: "Formatted at 2026-03-19 16:37:55 on dist-test-slave-svn8"
server_key: "fd9ef89c37b8354268d9b3700e13a0fb"
server_key_iv: "9ab415785ab9f979e587e2fc8b1a2609"
server_key_version: "encryptionkey@0"
I20260319 16:37:55.078990 14296 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:55.079766 14310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:55.079916 14296 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:37:55.079972 14296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal
uuid: "791286772f5e4f23852249e7d7bb8294"
format_stamp: "Formatted at 2026-03-19 16:37:55 on dist-test-slave-svn8"
server_key: "fd9ef89c37b8354268d9b3700e13a0fb"
server_key_iv: "9ab415785ab9f979e587e2fc8b1a2609"
server_key_version: "encryptionkey@0"
I20260319 16:37:55.080029 14296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:55.102563 14296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:55.102802 14296 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:55.102912 14296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:55.103099 14296 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260319 16:37:55.103428 14296 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260319 16:37:55.103462 14296 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:55.103492 14296 ts_tablet_manager.cc:616] Registered 0 tablets
I20260319 16:37:55.103508 14296 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:55.108708 14296 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.195:42813
I20260319 16:37:55.108776 14423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.195:42813 every 8 connection(s)
I20260319 16:37:55.109031 14296 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/data/info.pb
I20260319 16:37:55.117522 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.109700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.195:49763 (local address 127.24.187.254:42471)
0319 16:37:55.109776 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.109779 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.109977 (+ 198us) server_negotiation.cc:408] Connection header received
0319 16:37:55.110454 (+ 477us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.110456 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.110469 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.110540 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:55.110970 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.111469 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.112049 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.112180 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.117165 (+ 4985us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:55.117172 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:55.117172 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:55.117179 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:55.117203 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:55.117208 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:55.117302 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.117419 (+ 117us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.117456 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260319 16:37:55.117635 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.109665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:42471 (local address 127.24.187.195:49763)
0319 16:37:55.109859 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:55.109885 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:55.110351 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:55.110575 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:55.110579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:55.110589 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:37:55.110872 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.110875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.111566 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.111567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:55.111951 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.111953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.112056 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.117006 (+ 4950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0319 16:37:55.117007 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:37:55.117030 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0319 16:37:55.117050 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:37:55.117312 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:37:55.117316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:37:55.117318 (+ 2us) client_negotiation.cc:769] Sending connection context
0319 16:37:55.117374 (+ 56us) client_negotiation.cc:240] Negotiation successful
0319 16:37:55.117450 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260319 16:37:55.117781 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14296
I20260319 16:37:55.117885 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal/instance
I20260319 16:37:55.118083 25327 external_mini_cluster.cc:1468] Setting key d7b4d2b61d921f6842f3995a24398ad1
I20260319 16:37:55.118099 14424 heartbeater.cc:344] Connected to a master server at 127.24.187.254:42471
I20260319 16:37:55.118187 14424 heartbeater.cc:461] Registering TS with master...
I20260319 16:37:55.118445 14424 heartbeater.cc:507] Master 127.24.187.254:42471 requested a full tablet report, sending...
I20260319 16:37:55.118825 13974 ts_manager.cc:194] Registered new tserver with Master: 791286772f5e4f23852249e7d7bb8294 (127.24.187.195:42813)
I20260319 16:37:55.119179 25327 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260319 16:37:55.119421 13974 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.24.187.195:49763
I20260319 16:37:55.122944 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.120591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.24.187.254:42471)
0319 16:37:55.120634 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.120636 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.120756 (+ 120us) server_negotiation.cc:408] Connection header received
0319 16:37:55.120807 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.120808 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.120817 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.120861 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:55.121151 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.121616 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.122181 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.122349 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.122718 (+ 369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:55.122721 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:55.122722 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:55.122727 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:55.122740 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:55.122743 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:55.122790 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.122841 (+ 51us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.122874 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:37:55.124296 13974 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260319 16:37:55.125461 13974 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260319 16:37:55.130203 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.127314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.194:44893 (local address 127.0.0.1:37504)
0319 16:37:55.127636 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:55.127647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:55.127705 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:55.127936 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:55.127938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:55.127941 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:55.128167 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.128169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.129296 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.129297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:55.129931 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.129933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.130020 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.130053 (+ 33us) client_negotiation.cc:769] Sending connection context
0319 16:37:55.130091 (+ 38us) client_negotiation.cc:240] Negotiation successful
0319 16:37:55.130132 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260319 16:37:55.130854 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.127532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:43149 (local address 127.0.0.1:58748)
0319 16:37:55.127719 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:55.127727 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:55.127773 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:55.128099 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:55.128100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:55.128103 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:55.128293 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.128296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.129922 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.129923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:55.130661 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.130664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.130727 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.130730 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:55.130756 (+ 26us) client_negotiation.cc:240] Negotiation successful
0319 16:37:55.130790 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":57,"threads_started":1}
I20260319 16:37:55.131099 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.127419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.24.187.194:44893)
0319 16:37:55.127608 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.127615 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.127711 (+ 96us) server_negotiation.cc:408] Connection header received
0319 16:37:55.127775 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.127778 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.127793 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.127859 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:55.128252 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.129219 (+ 967us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.130412 (+ 1193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.130884 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.130964 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.130989 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.131026 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260319 16:37:55.131507 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.127474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.24.187.195:42813)
0319 16:37:55.127865 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.127873 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.127882 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:37:55.127916 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.127917 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.127932 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.127998 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:55.128304 (+ 306us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.129812 (+ 1508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.130893 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.131342 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.131405 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.131430 (+ 25us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.131457 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":78,"threads_started":1}
I20260319 16:37:55.131626 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.127720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.24.187.193:43149)
0319 16:37:55.127889 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.127897 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.127904 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:37:55.127938 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.127940 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.127963 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.128018 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:55.128376 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.129847 (+ 1471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.130736 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.131423 (+ 687us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.131496 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.131525 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.131559 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260319 16:37:55.131639 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.127314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42813 (local address 127.0.0.1:57720)
0319 16:37:55.127503 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:55.127531 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:55.127603 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:55.128006 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:55.128007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:55.128009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:55.128186 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.128187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.129910 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.129911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:55.130725 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.130727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.130791 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.130794 (+ 3us) client_negotiation.cc:769] Sending connection context
0319 16:37:55.130822 (+ 28us) client_negotiation.cc:240] Negotiation successful
0319 16:37:55.130856 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260319 16:37:55.131670 14227 tablet_service.cc:1511] Processing CreateTablet for tablet ecbe9814136641e695c7f940fb818abe (DEFAULT_TABLE table=test-table [id=77133f34009646cea6a8ed65804e9983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:55.131918 14227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbe9814136641e695c7f940fb818abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:55.132095 14096 tablet_service.cc:1511] Processing CreateTablet for tablet ecbe9814136641e695c7f940fb818abe (DEFAULT_TABLE table=test-table [id=77133f34009646cea6a8ed65804e9983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:55.132175 14358 tablet_service.cc:1511] Processing CreateTablet for tablet ecbe9814136641e695c7f940fb818abe (DEFAULT_TABLE table=test-table [id=77133f34009646cea6a8ed65804e9983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260319 16:37:55.132345 14096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbe9814136641e695c7f940fb818abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:55.132368 14358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbe9814136641e695c7f940fb818abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:55.134317 14438 tablet_bootstrap.cc:492] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Bootstrap starting.
I20260319 16:37:55.134621 14439 tablet_bootstrap.cc:492] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Bootstrap starting.
I20260319 16:37:55.134892 14438 tablet_bootstrap.cc:654] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:55.135140 14439 tablet_bootstrap.cc:654] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:55.135156 14438 log.cc:826] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:55.135186 14440 tablet_bootstrap.cc:492] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87: Bootstrap starting.
I20260319 16:37:55.135419 14439 log.cc:826] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:55.135704 14440 tablet_bootstrap.cc:654] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:55.135849 14438 tablet_bootstrap.cc:492] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: No bootstrap required, opened a new log
I20260319 16:37:55.135900 14438 ts_tablet_manager.cc:1403] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:55.136068 14440 log.cc:826] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:55.136691 14440 tablet_bootstrap.cc:492] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87: No bootstrap required, opened a new log
I20260319 16:37:55.136739 14440 ts_tablet_manager.cc:1403] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260319 16:37:55.136974 14438 raft_consensus.cc:359] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.137082 14438 raft_consensus.cc:385] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:55.137107 14438 raft_consensus.cc:740] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791286772f5e4f23852249e7d7bb8294, State: Initialized, Role: FOLLOWER
I20260319 16:37:55.137204 14438 consensus_queue.cc:260] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.137430 14438 ts_tablet_manager.cc:1434] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260319 16:37:55.137485 14424 heartbeater.cc:499] Master 127.24.187.254:42471 was elected leader, sending a full tablet report...
I20260319 16:37:55.138003 14439 tablet_bootstrap.cc:492] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: No bootstrap required, opened a new log
I20260319 16:37:55.138067 14439 ts_tablet_manager.cc:1403] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260319 16:37:55.138276 14440 raft_consensus.cc:359] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.138391 14440 raft_consensus.cc:385] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:55.138409 14440 raft_consensus.cc:740] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2960aa2f3f3d461e8efd14266f910f87, State: Initialized, Role: FOLLOWER
I20260319 16:37:55.138481 14440 consensus_queue.cc:260] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.138679 14440 ts_tablet_manager.cc:1434] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:55.138749 14162 heartbeater.cc:499] Master 127.24.187.254:42471 was elected leader, sending a full tablet report...
I20260319 16:37:55.139384 14439 raft_consensus.cc:359] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.139508 14439 raft_consensus.cc:385] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:55.139533 14439 raft_consensus.cc:740] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbeb4f96eb649c2bb61af6759af9041, State: Initialized, Role: FOLLOWER
I20260319 16:37:55.139613 14439 consensus_queue.cc:260] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.139793 14439 ts_tablet_manager.cc:1434] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260319 16:37:55.139858 14293 heartbeater.cc:499] Master 127.24.187.254:42471 was elected leader, sending a full tablet report...
W20260319 16:37:55.239982 14294 tablet.cc:2379] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:37:55.359531 14425 tablet.cc:2379] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260319 16:37:55.374341 14163 tablet.cc:2379] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260319 16:37:55.506021 14446 raft_consensus.cc:493] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260319 16:37:55.506158 14446 raft_consensus.cc:515] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.506505 14446 leader_election.cc:290] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791286772f5e4f23852249e7d7bb8294 (127.24.187.195:42813), 2960aa2f3f3d461e8efd14266f910f87 (127.24.187.193:43149)
I20260319 16:37:55.508761 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.506895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:43149 (local address 127.24.187.194:58813)
0319 16:37:55.506979 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:55.506989 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:55.507053 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:55.507192 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:55.507193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:55.507195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:55.507424 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.507426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.508025 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.508026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:55.508598 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.508599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.508653 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.508658 (+ 5us) client_negotiation.cc:769] Sending connection context
0319 16:37:55.508673 (+ 15us) client_negotiation.cc:240] Negotiation successful
0319 16:37:55.508698 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":38,"threads_started":1}
I20260319 16:37:55.509160 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.506711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.195:42813 (local address 127.24.187.194:57825)
0319 16:37:55.506825 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:55.506836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:37:55.506904 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:37:55.507128 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:37:55.507129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:37:55.507131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0319 16:37:55.507346 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.507348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.508267 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.508268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:37:55.508953 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:37:55.508955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.509017 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.509021 (+ 4us) client_negotiation.cc:769] Sending connection context
0319 16:37:55.509052 (+ 31us) client_negotiation.cc:240] Negotiation successful
0319 16:37:55.509087 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260319 16:37:55.509233 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.506953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:58813 (local address 127.24.187.193:43149)
0319 16:37:55.507004 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.507006 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.507040 (+ 34us) server_negotiation.cc:408] Connection header received
0319 16:37:55.507115 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.507117 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.507128 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.507178 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:55.507499 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.507954 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.508697 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.509120 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.509128 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.509148 (+ 20us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.509178 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:37:55.509500 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.506775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.187.194:57825 (local address 127.24.187.195:42813)
0319 16:37:55.506824 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.506825 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.506893 (+ 68us) server_negotiation.cc:408] Connection header received
0319 16:37:55.507002 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.507004 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.507018 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.507048 (+ 30us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0319 16:37:55.507424 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.507955 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.509034 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.509403 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.509410 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.509431 (+ 21us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.509455 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:37:55.509513 14116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecbe9814136641e695c7f940fb818abe" candidate_uuid: "ebbeb4f96eb649c2bb61af6759af9041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2960aa2f3f3d461e8efd14266f910f87" is_pre_election: true
I20260319 16:37:55.509651 14116 raft_consensus.cc:2468] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbeb4f96eb649c2bb61af6759af9041 in term 0.
I20260319 16:37:55.509895 14182 leader_election.cc:304] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2960aa2f3f3d461e8efd14266f910f87, ebbeb4f96eb649c2bb61af6759af9041; no voters:
I20260319 16:37:55.509955 14378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecbe9814136641e695c7f940fb818abe" candidate_uuid: "ebbeb4f96eb649c2bb61af6759af9041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791286772f5e4f23852249e7d7bb8294" is_pre_election: true
I20260319 16:37:55.510082 14378 raft_consensus.cc:2468] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbeb4f96eb649c2bb61af6759af9041 in term 0.
I20260319 16:37:55.510056 14446 raft_consensus.cc:2804] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260319 16:37:55.510115 14446 raft_consensus.cc:493] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260319 16:37:55.510145 14446 raft_consensus.cc:3060] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:55.510810 14446 raft_consensus.cc:515] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.510929 14446 leader_election.cc:290] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [CANDIDATE]: Term 1 election: Requested vote from peers 791286772f5e4f23852249e7d7bb8294 (127.24.187.195:42813), 2960aa2f3f3d461e8efd14266f910f87 (127.24.187.193:43149)
I20260319 16:37:55.511169 14378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecbe9814136641e695c7f940fb818abe" candidate_uuid: "ebbeb4f96eb649c2bb61af6759af9041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791286772f5e4f23852249e7d7bb8294"
I20260319 16:37:55.511212 14116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecbe9814136641e695c7f940fb818abe" candidate_uuid: "ebbeb4f96eb649c2bb61af6759af9041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2960aa2f3f3d461e8efd14266f910f87"
I20260319 16:37:55.511240 14378 raft_consensus.cc:3060] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:55.511284 14116 raft_consensus.cc:3060] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:55.511934 14116 raft_consensus.cc:2468] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbeb4f96eb649c2bb61af6759af9041 in term 1.
I20260319 16:37:55.511936 14378 raft_consensus.cc:2468] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbeb4f96eb649c2bb61af6759af9041 in term 1.
I20260319 16:37:55.512079 14182 leader_election.cc:304] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2960aa2f3f3d461e8efd14266f910f87, ebbeb4f96eb649c2bb61af6759af9041; no voters:
I20260319 16:37:55.512159 14446 raft_consensus.cc:2804] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:55.512319 14446 raft_consensus.cc:697] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 1 LEADER]: Becoming Leader. State: Replica: ebbeb4f96eb649c2bb61af6759af9041, State: Running, Role: LEADER
I20260319 16:37:55.512421 14446 consensus_queue.cc:237] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } }
I20260319 16:37:55.513235 13973 catalog_manager.cc:5671] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbeb4f96eb649c2bb61af6759af9041 (127.24.187.194). New cstate: current_term: 1 leader_uuid: "ebbeb4f96eb649c2bb61af6759af9041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbeb4f96eb649c2bb61af6759af9041" member_type: VOTER last_known_addr { host: "127.24.187.194" port: 44893 } health_report { overall_health: HEALTHY } } }
I20260319 16:37:55.573779 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.571815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.24.187.194:44893)
0319 16:37:55.571869 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:55.571871 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:55.571878 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:37:55.571919 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:55.571920 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:55.571938 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:55.571988 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:37:55.572187 (+ 199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.572713 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:55.573294 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:55.573433 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:55.573472 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:37:55.573569 (+ 97us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:37:55.573627 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:55.573658 (+ 31us) server_negotiation.cc:300] Negotiation successful
0319 16:37:55.573680 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:37:55.575384 14116 raft_consensus.cc:1275] T ecbe9814136641e695c7f940fb818abe P 2960aa2f3f3d461e8efd14266f910f87 [term 1 FOLLOWER]: Refusing update from remote peer ebbeb4f96eb649c2bb61af6759af9041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:55.575426 14378 raft_consensus.cc:1275] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 1 FOLLOWER]: Refusing update from remote peer ebbeb4f96eb649c2bb61af6759af9041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:37:55.575763 14446 consensus_queue.cc:1048] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791286772f5e4f23852249e7d7bb8294" member_type: VOTER last_known_addr { host: "127.24.187.195" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:55.575889 14449 consensus_queue.cc:1048] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2960aa2f3f3d461e8efd14266f910f87" member_type: VOTER last_known_addr { host: "127.24.187.193" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:37:55.579952 13974 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:55.580034 13974 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260319 16:37:55.580674 13974 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a856915d84c24eceb259553bbcf6c62e: Sending DeleteTablet for 3 replicas of tablet ecbe9814136641e695c7f940fb818abe
I20260319 16:37:55.580952 14096 tablet_service.cc:1558] Processing DeleteTablet for tablet ecbe9814136641e695c7f940fb818abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:55 UTC) from {username='slave'} at 127.0.0.1:58748
I20260319 16:37:55.580960 14358 tablet_service.cc:1558] Processing DeleteTablet for tablet ecbe9814136641e695c7f940fb818abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:55 UTC) from {username='slave'} at 127.0.0.1:57720
I20260319 16:37:55.581003 14227 tablet_service.cc:1558] Processing DeleteTablet for tablet ecbe9814136641e695c7f940fb818abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-19 16:37:55 UTC) from {username='slave'} at 127.0.0.1:37504
I20260319 16:37:55.581274 14464 tablet_replica.cc:333] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: stopping tablet replica
I20260319 16:37:55.581324 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14034
I20260319 16:37:55.581358 14464 raft_consensus.cc:2243] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 1 LEADER]: Raft consensus shutting down.
I20260319 16:37:55.581969 14464 raft_consensus.cc:2272] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:55.582351 14464 ts_tablet_manager.cc:1916] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:55.583783 14464 ts_tablet_manager.cc:1929] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:55.583827 14464 log.cc:1199] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-1/wal/wals/ecbe9814136641e695c7f940fb818abe
I20260319 16:37:55.583999 14464 ts_tablet_manager.cc:1950] T ecbe9814136641e695c7f940fb818abe P ebbeb4f96eb649c2bb61af6759af9041: Deleting consensus metadata
I20260319 16:37:55.584265 13961 catalog_manager.cc:5002] TS ebbeb4f96eb649c2bb61af6759af9041 (127.24.187.194:44893): tablet ecbe9814136641e695c7f940fb818abe (table test-table [id=77133f34009646cea6a8ed65804e9983]) successfully deleted
I20260319 16:37:55.585837 14462 tablet_replica.cc:333] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: stopping tablet replica
I20260319 16:37:55.585920 14462 raft_consensus.cc:2243] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260319 16:37:55.586005 14462 raft_consensus.cc:2272] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260319 16:37:55.586400 14462 ts_tablet_manager.cc:1916] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260319 16:37:55.587845 14462 ts_tablet_manager.cc:1929] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260319 16:37:55.587908 14462 log.cc:1199] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Deleting WAL directory at /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1773937967898507-25327-0/minicluster-data/ts-2/wal/wals/ecbe9814136641e695c7f940fb818abe
I20260319 16:37:55.588133 14462 ts_tablet_manager.cc:1950] T ecbe9814136641e695c7f940fb818abe P 791286772f5e4f23852249e7d7bb8294: Deleting consensus metadata
I20260319 16:37:55.588536 13960 catalog_manager.cc:5002] TS 791286772f5e4f23852249e7d7bb8294 (127.24.187.195:42813): tablet ecbe9814136641e695c7f940fb818abe (table test-table [id=77133f34009646cea6a8ed65804e9983]) successfully deleted
W20260319 16:37:55.589974 13960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.187.193:43149 (error 108)
I20260319 16:37:55.590256 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14165
I20260319 16:37:55.590440 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:55.590309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.193:43149 (local address 127.0.0.1:58756)
0319 16:37:55.590356 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0319 16:37:55.590404 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.193:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260319 16:37:55.590930 13960 catalog_manager.cc:4729] TS 2960aa2f3f3d461e8efd14266f910f87 (127.24.187.193:43149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecbe9814136641e695c7f940fb818abe: Network error: Client connection negotiation failed: client connection to 127.24.187.193:43149: connect: Connection refused (error 111)
I20260319 16:37:55.595326 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14296
I20260319 16:37:55.599398 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 13943
2026-03-19T16:37:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1019 ms)
[----------] 6 tests from EncryptionPolicyTest (11190 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:37:55 dist-test-slave-svn8 krb5kdc[14468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:37:55 dist-test-slave-svn8 krb5kdc[14468](info): set up 2 sockets
Mar 19 16:37:55 dist-test-slave-svn8 krb5kdc[14468](info): commencing operation
krb5kdc: starting...
W20260319 16:37:57.434338 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:37:57 dist-test-slave-svn8 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:37:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:57.564347 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:57.638121 14484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:57.638299 14484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:57.638329 14484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:57.639803 14484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:57.639849 14484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:57.639868 14484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:57.639882 14484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:57.639896 14484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:57.641510 14484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:57.641749 14484 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:57.641911 14484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:57.644083 14492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:57.644096 14490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:57.644125 14489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:57.644404 14484 server_base.cc:1060] running on GCE node
I20260319 16:37:57.644589 14484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:57.644781 14484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:57.645918 14484 hybrid_clock.cc:648] HybridClock initialized: now 1773938277645893 us; error 26 us; skew 500 ppm
Mar 19 16:37:57 dist-test-slave-svn8 krb5kdc[14468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:57.648226 14484 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:37:57.648721 14484 webserver.cc:492] Webserver started at http://127.0.0.1:40917/ using document root <none> and password file <none>
I20260319 16:37:57.648880 14484 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:57.648914 14484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:57.649010 14484 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:57.649981 14484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "500934750f474f849504e1cd1842d061"
format_stamp: "Formatted at 2026-03-19 16:37:57 on dist-test-slave-svn8"
server_key: "361a7caa001c1b927491a3efcf374f53"
server_key_iv: "8429628514a630f9c5f0d05006372056"
server_key_version: "encryptionkey@0"
I20260319 16:37:57.650272 14484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "500934750f474f849504e1cd1842d061"
format_stamp: "Formatted at 2026-03-19 16:37:57 on dist-test-slave-svn8"
server_key: "361a7caa001c1b927491a3efcf374f53"
server_key_iv: "8429628514a630f9c5f0d05006372056"
server_key_version: "encryptionkey@0"
I20260319 16:37:57.651348 14484 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:37:57.651954 14499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:57.652112 14484 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:37:57.652168 14484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "500934750f474f849504e1cd1842d061"
format_stamp: "Formatted at 2026-03-19 16:37:57 on dist-test-slave-svn8"
server_key: "361a7caa001c1b927491a3efcf374f53"
server_key_iv: "8429628514a630f9c5f0d05006372056"
server_key_version: "encryptionkey@0"
I20260319 16:37:57.652228 14484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:57.671080 14484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:57.672902 14484 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:57.673034 14484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:57.676796 14484 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42721
I20260319 16:37:57.676851 14551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42721 every 8 connection(s)
I20260319 16:37:57.677142 14484 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:57.677744 14552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:57.678637 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14484
I20260319 16:37:57.678740 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:57.678937 25327 external_mini_cluster.cc:1468] Setting key 1c3056802a3631b85ebb89c5e51d6579
I20260319 16:37:57.680644 14552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061: Bootstrap starting.
I20260319 16:37:57.681160 14552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:57.681397 14552 log.cc:826] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:57.681962 14552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061: No bootstrap required, opened a new log
I20260319 16:37:57.683092 14552 raft_consensus.cc:359] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500934750f474f849504e1cd1842d061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42721 } }
I20260319 16:37:57.683209 14552 raft_consensus.cc:385] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:57.683236 14552 raft_consensus.cc:740] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500934750f474f849504e1cd1842d061, State: Initialized, Role: FOLLOWER
I20260319 16:37:57.683326 14552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500934750f474f849504e1cd1842d061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42721 } }
I20260319 16:37:57.683396 14552 raft_consensus.cc:399] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:57.683424 14552 raft_consensus.cc:493] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:57.683466 14552 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:57.684111 14552 raft_consensus.cc:515] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500934750f474f849504e1cd1842d061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42721 } }
I20260319 16:37:57.684222 14552 leader_election.cc:304] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500934750f474f849504e1cd1842d061; no voters:
Mar 19 16:37:57 dist-test-slave-svn8 krb5kdc[14468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260319 16:37:57.684371 14552 leader_election.cc:290] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:57.684527 14558 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:57.684602 14552 sys_catalog.cc:565] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:57.684680 14558 raft_consensus.cc:697] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [term 1 LEADER]: Becoming Leader. State: Replica: 500934750f474f849504e1cd1842d061, State: Running, Role: LEADER
I20260319 16:37:57.684759 14558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500934750f474f849504e1cd1842d061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42721 } }
I20260319 16:37:57.685030 14558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500934750f474f849504e1cd1842d061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500934750f474f849504e1cd1842d061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42721 } } }
I20260319 16:37:57.685110 14558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:57.685518 14558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500934750f474f849504e1cd1842d061. Latest consensus state: current_term: 1 leader_uuid: "500934750f474f849504e1cd1842d061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500934750f474f849504e1cd1842d061" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42721 } } }
I20260319 16:37:57.685581 14558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061 [sys.catalog]: This master's current role is: LEADER
W20260319 16:37:57.686337 14573 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:37:57.686389 14573 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:37:57.686440 14566 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:57.686606 14566 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:57.687814 14566 catalog_manager.cc:1357] Generated new cluster ID: 68f669b0464b43f2b41788710423db41
I20260319 16:37:57.687853 14566 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:37:57.688864 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:57.679486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48341 (local address 127.0.0.1:42721)
0319 16:37:57.679664 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:57.679675 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:57.679705 (+ 30us) server_negotiation.cc:408] Connection header received
0319 16:37:57.680075 (+ 370us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:57.680081 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:57.680120 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:57.680221 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:57.680962 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:57.681786 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:57.682348 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:57.682602 (+ 254us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:57.684890 (+ 2288us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:57.684898 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:57.684899 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:57.684915 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:57.687388 (+ 2473us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:57.687821 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:57.687824 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:57.687825 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:57.687878 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:57.688092 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:57.688093 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:57.688094 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:57.688306 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:57.688450 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:57.688616 (+ 166us) server_negotiation.cc:300] Negotiation successful
0319 16:37:57.688711 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260319 16:37:57.689911 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:37:57.697671 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:57.691277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36369 (local address 127.0.0.1:42721)
0319 16:37:57.691309 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:57.691311 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:57.691489 (+ 178us) server_negotiation.cc:408] Connection header received
0319 16:37:57.691574 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:57.691575 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:57.691583 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:57.691607 (+ 24us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:57.692372 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:57.693063 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:57.693787 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:57.693952 (+ 165us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:57.695225 (+ 1273us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:57.695228 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:57.695228 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:37:57.695234 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:57.696628 (+ 1394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:57.696959 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:57.696960 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:57.696960 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:57.696980 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:37:57.697145 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:37:57.697146 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:37:57.697147 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:37:57.697247 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:57.697312 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:57.697515 (+ 203us) server_negotiation.cc:300] Negotiation successful
0319 16:37:57.697582 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260319 16:37:57.705214 14566 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:57.705961 14566 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:57.715159 14566 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 500934750f474f849504e1cd1842d061: Generated new TSK 0
I20260319 16:37:57.715348 14566 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:37:57.717898 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14484
W20260319 16:37:57.722313 14466 connection.cc:537] client connection to 127.0.0.1:42721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42721): Connection reset by peer (error 104)
2026-03-19T16:37:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2117 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:37:57 dist-test-slave-svn8 krb5kdc[14583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:37:57 dist-test-slave-svn8 krb5kdc[14583](info): set up 2 sockets
Mar 19 16:37:57 dist-test-slave-svn8 krb5kdc[14583](info): commencing operation
krb5kdc: starting...
W20260319 16:37:59.554678 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:37:59 dist-test-slave-svn8 krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:37:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:37:59.688004 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:37:59.762089 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:37:59.762243 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:37:59.762262 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:37:59.763715 14599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:37:59.763757 14599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:37:59.763769 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:37:59.763779 14599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:37:59.763790 14599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:37:59.765394 14599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:37:59.765585 14599 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:37:59.765758 14599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:37:59.767884 14604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:59.767863 14605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:37:59.767884 14607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:37:59.768059 14599 server_base.cc:1060] running on GCE node
I20260319 16:37:59.768272 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:37:59.768522 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:37:59.769651 14599 hybrid_clock.cc:648] HybridClock initialized: now 1773938279769630 us; error 31 us; skew 500 ppm
Mar 19 16:37:59 dist-test-slave-svn8 krb5kdc[14583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:37:59.771853 14599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:37:59.772298 14599 webserver.cc:492] Webserver started at http://127.0.0.1:45063/ using document root <none> and password file <none>
I20260319 16:37:59.772459 14599 fs_manager.cc:362] Metadata directory not provided
I20260319 16:37:59.772493 14599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:37:59.772593 14599 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:37:59.773512 14599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "4e561adc45f14d24b3a37e02cf5f6ac3"
format_stamp: "Formatted at 2026-03-19 16:37:59 on dist-test-slave-svn8"
server_key: "8af827deaadf308638cec79c2a8ea00e"
server_key_iv: "3ef987a41d084c3b60cfab88311d0a17"
server_key_version: "encryptionkey@0"
I20260319 16:37:59.773862 14599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "4e561adc45f14d24b3a37e02cf5f6ac3"
format_stamp: "Formatted at 2026-03-19 16:37:59 on dist-test-slave-svn8"
server_key: "8af827deaadf308638cec79c2a8ea00e"
server_key_iv: "3ef987a41d084c3b60cfab88311d0a17"
server_key_version: "encryptionkey@0"
I20260319 16:37:59.774957 14599 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:37:59.775550 14614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:37:59.775718 14599 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:37:59.775776 14599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "4e561adc45f14d24b3a37e02cf5f6ac3"
format_stamp: "Formatted at 2026-03-19 16:37:59 on dist-test-slave-svn8"
server_key: "8af827deaadf308638cec79c2a8ea00e"
server_key_iv: "3ef987a41d084c3b60cfab88311d0a17"
server_key_version: "encryptionkey@0"
I20260319 16:37:59.775835 14599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:37:59.783232 14599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:37:59.785266 14599 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:37:59.785399 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:37:59.789067 14599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39525
I20260319 16:37:59.789101 14666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:39525 every 8 connection(s)
I20260319 16:37:59.789376 14599 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:37:59.789944 14667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:37:59.792243 14667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3: Bootstrap starting.
I20260319 16:37:59.792274 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14599
I20260319 16:37:59.792379 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:37:59.792536 25327 external_mini_cluster.cc:1468] Setting key a0d20df480f51aac12e4edb600a48a24
I20260319 16:37:59.792809 14667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3: Neither blocks nor log segments found. Creating new log.
I20260319 16:37:59.793090 14667 log.cc:826] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3: Log is configured to *not* fsync() on all Append() calls
I20260319 16:37:59.793756 14667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3: No bootstrap required, opened a new log
I20260319 16:37:59.794884 14667 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260319 16:37:59.795014 14667 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:37:59.795045 14667 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e561adc45f14d24b3a37e02cf5f6ac3, State: Initialized, Role: FOLLOWER
I20260319 16:37:59.795138 14667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260319 16:37:59.795193 14667 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:37:59.795217 14667 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:37:59.795256 14667 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:37:59.795945 14667 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260319 16:37:59.796108 14667 leader_election.cc:304] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e561adc45f14d24b3a37e02cf5f6ac3; no voters:
I20260319 16:37:59.796277 14667 leader_election.cc:290] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:37:59.796363 14672 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:37:59.796514 14672 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [term 1 LEADER]: Becoming Leader. State: Replica: 4e561adc45f14d24b3a37e02cf5f6ac3, State: Running, Role: LEADER
I20260319 16:37:59.796545 14667 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:37:59.796979 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:59.793039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44253 (local address 127.0.0.1:39525)
0319 16:37:59.793186 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:59.793197 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:59.793220 (+ 23us) server_negotiation.cc:408] Connection header received
0319 16:37:59.793570 (+ 350us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:59.793576 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:59.793616 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:59.793763 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:59.794175 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:59.794896 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:59.795640 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:59.795868 (+ 228us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:59.796291 (+ 423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:59.796299 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:59.796300 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:59.796502 (+ 202us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:59.796523 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:59.796532 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:59.796687 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:59.796811 (+ 124us) server_negotiation.cc:300] Negotiation successful
0319 16:37:59.796836 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260319 16:37:59.797863 14672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260319 16:37:59.797946 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:37:59.798354 14673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e561adc45f14d24b3a37e02cf5f6ac3. Latest consensus state: current_term: 1 leader_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } } }
I20260319 16:37:59.798390 14674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e561adc45f14d24b3a37e02cf5f6ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } } }
I20260319 16:37:59.799414 14673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [sys.catalog]: This master's current role is: LEADER
I20260319 16:37:59.799440 14674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3 [sys.catalog]: This master's current role is: LEADER
W20260319 16:37:59.798960 14686 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:37:59.799542 14686 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:37:59.799667 14693 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:37:59.799861 14693 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:37:59.801178 14693 catalog_manager.cc:1357] Generated new cluster ID: 2ae23ffe715e438ead7e78902dde0582
I20260319 16:37:59.801227 14693 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:37:59.802553 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:37:59.799345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34463 (local address 127.0.0.1:39525)
0319 16:37:59.799382 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:37:59.799384 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:37:59.799762 (+ 378us) server_negotiation.cc:408] Connection header received
0319 16:37:59.799850 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:37:59.799851 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:37:59.799861 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:37:59.799888 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:37:59.800253 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:59.800770 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:37:59.801433 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:37:59.801593 (+ 160us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:37:59.802243 (+ 650us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:37:59.802246 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:37:59.802246 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:37:59.802250 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:37:59.802262 (+ 12us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:37:59.802266 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:37:59.802297 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0319 16:37:59.802462 (+ 165us) server_negotiation.cc:300] Negotiation successful
0319 16:37:59.802472 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:37:59.823076 14693 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:37:59.823587 14693 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:37:59.834035 14693 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4e561adc45f14d24b3a37e02cf5f6ac3: Generated new TSK 0
I20260319 16:37:59.834194 14693 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:37:59.840806 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14599
W20260319 16:37:59.844996 14581 connection.cc:537] client connection to 127.0.0.1:39525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39525): Connection reset by peer (error 104)
2026-03-19T16:37:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:37:59 dist-test-slave-svn8 krb5kdc[14698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:37:59 dist-test-slave-svn8 krb5kdc[14698](info): set up 2 sockets
Mar 19 16:37:59 dist-test-slave-svn8 krb5kdc[14698](info): commencing operation
krb5kdc: starting...
W20260319 16:38:01.670385 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.812s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:01 dist-test-slave-svn8 krb5kdc[14698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:01.817458 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:01.896445 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:01.896673 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:01.896711 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:01.898332 14714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:01.898386 14714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:01.898403 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:01.898419 14714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:01.898434 14714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:01.900173 14714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:01.900414 14714 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:01.900596 14714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:01.902902 14720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:01.902912 14722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:01.902992 14719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:01.903123 14714 server_base.cc:1060] running on GCE node
I20260319 16:38:01.903388 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:01.903615 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:01.904762 14714 hybrid_clock.cc:648] HybridClock initialized: now 1773938281904740 us; error 35 us; skew 500 ppm
Mar 19 16:38:01 dist-test-slave-svn8 krb5kdc[14698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:01.907351 14714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:01.907863 14714 webserver.cc:492] Webserver started at http://127.0.0.1:38577/ using document root <none> and password file <none>
I20260319 16:38:01.908030 14714 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:01.908063 14714 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:01.908164 14714 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:01.909171 14714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "57189c91d1a44f8fae8f9c236b84295d"
format_stamp: "Formatted at 2026-03-19 16:38:01 on dist-test-slave-svn8"
server_key: "808d5c56f974b85743dab7fe8b561405"
server_key_iv: "bad55ece297c9c45b0b83ab5b4492d23"
server_key_version: "encryptionkey@0"
I20260319 16:38:01.909519 14714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "57189c91d1a44f8fae8f9c236b84295d"
format_stamp: "Formatted at 2026-03-19 16:38:01 on dist-test-slave-svn8"
server_key: "808d5c56f974b85743dab7fe8b561405"
server_key_iv: "bad55ece297c9c45b0b83ab5b4492d23"
server_key_version: "encryptionkey@0"
I20260319 16:38:01.910719 14714 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:38:01.911425 14729 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:01.911628 14714 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:38:01.911688 14714 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "57189c91d1a44f8fae8f9c236b84295d"
format_stamp: "Formatted at 2026-03-19 16:38:01 on dist-test-slave-svn8"
server_key: "808d5c56f974b85743dab7fe8b561405"
server_key_iv: "bad55ece297c9c45b0b83ab5b4492d23"
server_key_version: "encryptionkey@0"
I20260319 16:38:01.911752 14714 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:01.915097 14714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:01.922875 14714 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:01.923009 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:01.926853 14714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44211
I20260319 16:38:01.926901 14781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:44211 every 8 connection(s)
I20260319 16:38:01.927174 14714 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:01.927726 14782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:01.930146 14782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d: Bootstrap starting.
I20260319 16:38:01.930717 14782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:01.930935 14782 log.cc:826] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:01.931730 14782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d: No bootstrap required, opened a new log
I20260319 16:38:01.932618 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14714
I20260319 16:38:01.932727 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:01.932889 25327 external_mini_cluster.cc:1468] Setting key aaa7767cd35e927d69f09dd4a17c3e2f
I20260319 16:38:01.933063 14782 raft_consensus.cc:359] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57189c91d1a44f8fae8f9c236b84295d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260319 16:38:01.933231 14782 raft_consensus.cc:385] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:01.933266 14782 raft_consensus.cc:740] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57189c91d1a44f8fae8f9c236b84295d, State: Initialized, Role: FOLLOWER
I20260319 16:38:01.933382 14782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57189c91d1a44f8fae8f9c236b84295d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260319 16:38:01.933455 14782 raft_consensus.cc:399] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:01.933506 14782 raft_consensus.cc:493] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:01.933593 14782 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:01.934329 14782 raft_consensus.cc:515] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57189c91d1a44f8fae8f9c236b84295d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260319 16:38:01.934422 14782 leader_election.cc:304] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57189c91d1a44f8fae8f9c236b84295d; no voters:
I20260319 16:38:01.934584 14782 leader_election.cc:290] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:01.934615 14787 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:01.934789 14787 raft_consensus.cc:697] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [term 1 LEADER]: Becoming Leader. State: Replica: 57189c91d1a44f8fae8f9c236b84295d, State: Running, Role: LEADER
I20260319 16:38:01.934871 14782 sys_catalog.cc:565] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:01.934928 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:01.933412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59053 (local address 127.0.0.1:44211)
0319 16:38:01.933584 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:01.933595 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:01.933619 (+ 24us) server_negotiation.cc:408] Connection header received
0319 16:38:01.933748 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:01.933755 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:01.933783 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:01.933910 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:01.934499 (+ 589us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:01.934502 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:01.934503 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:01.934507 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:01.934531 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:01.934538 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:01.934613 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:01.934704 (+ 91us) server_negotiation.cc:300] Negotiation successful
0319 16:38:01.934741 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260319 16:38:01.934930 14787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57189c91d1a44f8fae8f9c236b84295d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260319 16:38:01.935353 14789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57189c91d1a44f8fae8f9c236b84295d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57189c91d1a44f8fae8f9c236b84295d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } } }
I20260319 16:38:01.935441 14789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:01.935653 14788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57189c91d1a44f8fae8f9c236b84295d. Latest consensus state: current_term: 1 leader_uuid: "57189c91d1a44f8fae8f9c236b84295d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57189c91d1a44f8fae8f9c236b84295d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } } }
I20260319 16:38:01.935775 14788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:01.935868 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:01.935911 14794 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:01.936426 14794 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:01.937989 14794 catalog_manager.cc:1357] Generated new cluster ID: 67414a342b3f4c0d9cdc64bbdcd0ad25
I20260319 16:38:01.938033 14794 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:01.938474 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:01.937354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57843 (local address 127.0.0.1:44211)
0319 16:38:01.937388 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:01.937389 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:01.937523 (+ 134us) server_negotiation.cc:408] Connection header received
0319 16:38:01.937554 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:01.937555 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:01.937567 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:01.937628 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:01.938249 (+ 621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:01.938251 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:01.938251 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:01.938255 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:01.938271 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:01.938274 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:01.938329 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:01.938403 (+ 74us) server_negotiation.cc:300] Negotiation successful
0319 16:38:01.938415 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260319 16:38:01.956053 14794 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:01.956238 14794 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:01.965202 14794 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 57189c91d1a44f8fae8f9c236b84295d: Generated new TSK 0
I20260319 16:38:01.965415 14794 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:01.969040 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14714
2026-03-19T16:38:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2128 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:02 dist-test-slave-svn8 krb5kdc[14813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:02 dist-test-slave-svn8 krb5kdc[14813](info): set up 2 sockets
Mar 19 16:38:02 dist-test-slave-svn8 krb5kdc[14813](info): commencing operation
krb5kdc: starting...
W20260319 16:38:03.818459 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.831s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:03 dist-test-slave-svn8 krb5kdc[14813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:03.957057 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:04.032138 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:04.032323 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:04.032366 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:04.033914 14829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:04.033962 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:04.033979 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:04.033995 14829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:04.034009 14829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:04.035678 14829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:04.035900 14829 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:04.036063 14829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:04.038408 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:04.038425 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:04.038408 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:04.038695 14829 server_base.cc:1060] running on GCE node
I20260319 16:38:04.038954 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:04.039165 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:04.040324 14829 hybrid_clock.cc:648] HybridClock initialized: now 1773938284040306 us; error 31 us; skew 500 ppm
Mar 19 16:38:04 dist-test-slave-svn8 krb5kdc[14813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:04.042826 14829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:04.043339 14829 webserver.cc:492] Webserver started at http://127.0.0.1:37653/ using document root <none> and password file <none>
I20260319 16:38:04.043509 14829 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:04.043545 14829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:04.043632 14829 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:04.044556 14829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "c08c5052628047a4a5022dbae6f61fd6"
format_stamp: "Formatted at 2026-03-19 16:38:04 on dist-test-slave-svn8"
server_key: "aadf49816944e0284ee2a36995e40ffd"
server_key_iv: "37d6bd80660b60b5cb9629397344417e"
server_key_version: "encryptionkey@0"
I20260319 16:38:04.044850 14829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "c08c5052628047a4a5022dbae6f61fd6"
format_stamp: "Formatted at 2026-03-19 16:38:04 on dist-test-slave-svn8"
server_key: "aadf49816944e0284ee2a36995e40ffd"
server_key_iv: "37d6bd80660b60b5cb9629397344417e"
server_key_version: "encryptionkey@0"
I20260319 16:38:04.046026 14829 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:38:04.046825 14844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:04.047027 14829 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:38:04.047086 14829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "c08c5052628047a4a5022dbae6f61fd6"
format_stamp: "Formatted at 2026-03-19 16:38:04 on dist-test-slave-svn8"
server_key: "aadf49816944e0284ee2a36995e40ffd"
server_key_iv: "37d6bd80660b60b5cb9629397344417e"
server_key_version: "encryptionkey@0"
I20260319 16:38:04.047151 14829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:04.061256 14829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:04.063344 14829 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:04.063453 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:04.067332 14829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34053
I20260319 16:38:04.067391 14896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34053 every 8 connection(s)
I20260319 16:38:04.067656 14829 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:04.068185 14897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:04.070600 14897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6: Bootstrap starting.
I20260319 16:38:04.071159 14897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:04.071394 14897 log.cc:826] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:04.071776 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14829
I20260319 16:38:04.071895 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:04.072075 25327 external_mini_cluster.cc:1468] Setting key 80f563ab436eca0264c88943bfce25d7
I20260319 16:38:04.072106 14897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6: No bootstrap required, opened a new log
I20260319 16:38:04.073967 14897 raft_consensus.cc:359] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c5052628047a4a5022dbae6f61fd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260319 16:38:04.074119 14897 raft_consensus.cc:385] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:04.074146 14897 raft_consensus.cc:740] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08c5052628047a4a5022dbae6f61fd6, State: Initialized, Role: FOLLOWER
I20260319 16:38:04.074249 14897 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c5052628047a4a5022dbae6f61fd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260319 16:38:04.074324 14897 raft_consensus.cc:399] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:04.074361 14897 raft_consensus.cc:493] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:04.074404 14897 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:04.075145 14897 raft_consensus.cc:515] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c5052628047a4a5022dbae6f61fd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260319 16:38:04.075261 14897 leader_election.cc:304] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08c5052628047a4a5022dbae6f61fd6; no voters:
I20260319 16:38:04.075421 14897 leader_election.cc:290] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:04.075498 14902 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:04.075646 14902 raft_consensus.cc:697] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [term 1 LEADER]: Becoming Leader. State: Replica: c08c5052628047a4a5022dbae6f61fd6, State: Running, Role: LEADER
I20260319 16:38:04.075661 14897 sys_catalog.cc:565] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:04.075790 14902 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c5052628047a4a5022dbae6f61fd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260319 16:38:04.076500 14903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08c5052628047a4a5022dbae6f61fd6. Latest consensus state: current_term: 1 leader_uuid: "c08c5052628047a4a5022dbae6f61fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c5052628047a4a5022dbae6f61fd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260319 16:38:04.076947 14903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:04.077157 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08c5052628047a4a5022dbae6f61fd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08c5052628047a4a5022dbae6f61fd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260319 16:38:04.077165 14917 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:04.077324 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:04.077746 14917 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Mar 19 16:38:04 dist-test-slave-svn8 krb5kdc[14813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260319 16:38:04.082151 14917 catalog_manager.cc:1357] Generated new cluster ID: d2abb11e47e742e2b58a82f6e3ae094c
I20260319 16:38:04.082221 14917 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:04.082707 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:04.072571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59385 (local address 127.0.0.1:34053)
0319 16:38:04.072723 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:04.072732 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:04.072750 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:38:04.073146 (+ 396us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:04.073152 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:04.073192 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:04.073288 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:04.074322 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:04.075436 (+ 1114us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:04.076170 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:04.076436 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:04.078971 (+ 2535us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:04.078978 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:04.078979 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:04.078997 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:04.080876 (+ 1879us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:04.081318 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:04.081322 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:04.081322 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:04.081384 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:04.081583 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:04.081584 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:04.081584 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:04.081865 (+ 281us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:04.082002 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:04.082430 (+ 428us) server_negotiation.cc:300] Negotiation successful
0319 16:38:04.082548 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260319 16:38:04.083535 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:04.087285 14917 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:04.087772 14917 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:04.091754 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:04.085044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55271 (local address 127.0.0.1:34053)
0319 16:38:04.085081 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:04.085082 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:04.085232 (+ 150us) server_negotiation.cc:408] Connection header received
0319 16:38:04.085273 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:04.085274 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:04.085285 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:04.085352 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:04.086086 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:04.086781 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:04.087583 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:04.087764 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:04.089164 (+ 1400us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:04.089168 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:04.089168 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:04.089176 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:04.090636 (+ 1460us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:04.090989 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:04.090990 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:04.090990 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:04.091013 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:04.091163 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:04.091164 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:04.091164 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:04.091264 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:04.091331 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:04.091544 (+ 213us) server_negotiation.cc:300] Negotiation successful
0319 16:38:04.091647 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:04.096566 14917 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c08c5052628047a4a5022dbae6f61fd6: Generated new TSK 0
I20260319 16:38:04.096807 14917 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:04.099697 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14829
2026-03-19T16:38:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2130 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:04 dist-test-slave-svn8 krb5kdc[14928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:04 dist-test-slave-svn8 krb5kdc[14928](info): set up 2 sockets
Mar 19 16:38:04 dist-test-slave-svn8 krb5kdc[14928](info): commencing operation
krb5kdc: starting...
W20260319 16:38:05.950336 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.832s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:06 dist-test-slave-svn8 krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:06.084056 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:06.158277 14944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:06.158456 14944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:06.158479 14944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:06.159920 14944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:06.159958 14944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:06.159971 14944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:06.159982 14944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:06.159992 14944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:06.161566 14944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:06.161751 14944 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:06.161895 14944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:06.164052 14952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:06.164034 14950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:06.164150 14949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:06.164351 14944 server_base.cc:1060] running on GCE node
I20260319 16:38:06.164532 14944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:06.164749 14944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:06.165874 14944 hybrid_clock.cc:648] HybridClock initialized: now 1773938286165861 us; error 22 us; skew 500 ppm
Mar 19 16:38:06 dist-test-slave-svn8 krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:06.168010 14944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:06.168499 14944 webserver.cc:492] Webserver started at http://127.0.0.1:33793/ using document root <none> and password file <none>
I20260319 16:38:06.168660 14944 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:06.168694 14944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:06.168792 14944 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:06.169662 14944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "01a103c881b04f238b774331abe8772b"
format_stamp: "Formatted at 2026-03-19 16:38:06 on dist-test-slave-svn8"
server_key: "b5027cf5f4b15518452b414204c29429"
server_key_iv: "70b5d8ce1383b6d09915996bdedd9c42"
server_key_version: "encryptionkey@0"
I20260319 16:38:06.169981 14944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "01a103c881b04f238b774331abe8772b"
format_stamp: "Formatted at 2026-03-19 16:38:06 on dist-test-slave-svn8"
server_key: "b5027cf5f4b15518452b414204c29429"
server_key_iv: "70b5d8ce1383b6d09915996bdedd9c42"
server_key_version: "encryptionkey@0"
I20260319 16:38:06.171046 14944 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:38:06.171638 14959 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:06.171789 14944 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:38:06.171844 14944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "01a103c881b04f238b774331abe8772b"
format_stamp: "Formatted at 2026-03-19 16:38:06 on dist-test-slave-svn8"
server_key: "b5027cf5f4b15518452b414204c29429"
server_key_iv: "70b5d8ce1383b6d09915996bdedd9c42"
server_key_version: "encryptionkey@0"
I20260319 16:38:06.171901 14944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:06.184947 14944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:06.186928 14944 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:06.187058 14944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:06.190851 14944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46063
I20260319 16:38:06.190887 15011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:46063 every 8 connection(s)
I20260319 16:38:06.191165 14944 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:06.191704 15012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:06.194089 15012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b: Bootstrap starting.
I20260319 16:38:06.194618 15012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:06.194815 15012 log.cc:826] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:06.195444 15012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b: No bootstrap required, opened a new log
I20260319 16:38:06.196591 15012 raft_consensus.cc:359] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a103c881b04f238b774331abe8772b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260319 16:38:06.196708 15012 raft_consensus.cc:385] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:06.196728 15012 raft_consensus.cc:740] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a103c881b04f238b774331abe8772b, State: Initialized, Role: FOLLOWER
I20260319 16:38:06.196830 15012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a103c881b04f238b774331abe8772b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260319 16:38:06.196895 15012 raft_consensus.cc:399] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:06.196926 15012 raft_consensus.cc:493] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:06.196956 15012 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:06.197459 15012 raft_consensus.cc:515] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a103c881b04f238b774331abe8772b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260319 16:38:06.197563 15012 leader_election.cc:304] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01a103c881b04f238b774331abe8772b; no voters:
I20260319 16:38:06.197680 15012 leader_election.cc:290] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:06.197800 15015 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:06.197912 15012 sys_catalog.cc:565] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:06.197957 15015 raft_consensus.cc:697] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [term 1 LEADER]: Becoming Leader. State: Replica: 01a103c881b04f238b774331abe8772b, State: Running, Role: LEADER
I20260319 16:38:06.198073 15015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a103c881b04f238b774331abe8772b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260319 16:38:06.198441 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 14944
I20260319 16:38:06.198565 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:06.198761 25327 external_mini_cluster.cc:1468] Setting key 9f2856dfde9b7f326f016b682ee8be03
I20260319 16:38:06.199247 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01a103c881b04f238b774331abe8772b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a103c881b04f238b774331abe8772b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } } }
I20260319 16:38:06.199354 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:06.199635 15017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a103c881b04f238b774331abe8772b. Latest consensus state: current_term: 1 leader_uuid: "01a103c881b04f238b774331abe8772b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a103c881b04f238b774331abe8772b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } } }
I20260319 16:38:06.199705 15017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b [sys.catalog]: This master's current role is: LEADER
W20260319 16:38:06.199954 15031 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:06.199998 15031 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:06.200410 15032 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:06.200605 15032 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:06.202023 15032 catalog_manager.cc:1357] Generated new cluster ID: 1000bf2ae2e24d57a2b065546f54df5b
I20260319 16:38:06.202075 15032 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:06.203415 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:06.199381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35065 (local address 127.0.0.1:46063)
0319 16:38:06.199672 (+ 291us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:06.199682 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:06.199703 (+ 21us) server_negotiation.cc:408] Connection header received
0319 16:38:06.199808 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:06.199814 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:06.199859 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:06.200577 (+ 718us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:06.200702 (+ 125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:06.201522 (+ 820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:06.202276 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:06.202543 (+ 267us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:06.202832 (+ 289us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:06.202837 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:06.202837 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:06.203060 (+ 223us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:06.203081 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:06.203090 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:06.203165 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:06.203224 (+ 59us) server_negotiation.cc:300] Negotiation successful
0319 16:38:06.203270 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":76,"threads_started":1}
I20260319 16:38:06.204260 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:06.212663 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:06.206025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51945 (local address 127.0.0.1:46063)
0319 16:38:06.206059 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:06.206061 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:06.209897 (+ 3836us) server_negotiation.cc:408] Connection header received
0319 16:38:06.209959 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:06.209960 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:06.209974 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:06.210030 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:06.210463 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:06.211171 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:06.211866 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:06.212052 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:06.212407 (+ 355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:06.212410 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:06.212411 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:06.212416 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:06.212429 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:06.212433 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:06.212486 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:06.212548 (+ 62us) server_negotiation.cc:300] Negotiation successful
0319 16:38:06.212578 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260319 16:38:06.232175 15032 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:06.232745 15032 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:06.241415 15032 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 01a103c881b04f238b774331abe8772b: Generated new TSK 0
I20260319 16:38:06.241621 15032 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:06.250232 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 14944
2026-03-19T16:38:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2150 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:06 dist-test-slave-svn8 krb5kdc[15043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:06 dist-test-slave-svn8 krb5kdc[15043](info): set up 2 sockets
Mar 19 16:38:06 dist-test-slave-svn8 krb5kdc[15043](info): commencing operation
krb5kdc: starting...
W20260319 16:38:08.090302 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:08 dist-test-slave-svn8 krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:08.221487 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:08.295791 15059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:08.295959 15059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:08.295991 15059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:08.297468 15059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:08.297513 15059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:08.297531 15059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:08.297546 15059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:08.297561 15059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:08.299256 15059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:08.299485 15059 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:08.299634 15059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:08.301853 15064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:08.301858 15067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:08.301862 15065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:08.302045 15059 server_base.cc:1060] running on GCE node
I20260319 16:38:08.302235 15059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:08.302501 15059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:08.303629 15059 hybrid_clock.cc:648] HybridClock initialized: now 1773938288303618 us; error 27 us; skew 500 ppm
Mar 19 16:38:08 dist-test-slave-svn8 krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:08.306159 15059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:08.306692 15059 webserver.cc:492] Webserver started at http://127.0.0.1:38955/ using document root <none> and password file <none>
I20260319 16:38:08.306852 15059 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:08.306885 15059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:08.306982 15059 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:08.307893 15059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "304dca8e4ff64d0796f0b35878f5722d"
format_stamp: "Formatted at 2026-03-19 16:38:08 on dist-test-slave-svn8"
server_key: "5cafa25bc6893701d857d10bb5c3e4de"
server_key_iv: "fbeb312dea4d46189d4ad398fca95d22"
server_key_version: "encryptionkey@0"
I20260319 16:38:08.308172 15059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "304dca8e4ff64d0796f0b35878f5722d"
format_stamp: "Formatted at 2026-03-19 16:38:08 on dist-test-slave-svn8"
server_key: "5cafa25bc6893701d857d10bb5c3e4de"
server_key_iv: "fbeb312dea4d46189d4ad398fca95d22"
server_key_version: "encryptionkey@0"
I20260319 16:38:08.309335 15059 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:38:08.310125 15074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:08.310381 15059 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:38:08.310443 15059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "304dca8e4ff64d0796f0b35878f5722d"
format_stamp: "Formatted at 2026-03-19 16:38:08 on dist-test-slave-svn8"
server_key: "5cafa25bc6893701d857d10bb5c3e4de"
server_key_iv: "fbeb312dea4d46189d4ad398fca95d22"
server_key_version: "encryptionkey@0"
I20260319 16:38:08.310506 15059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:08.313768 15059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:08.320020 15059 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:08.320147 15059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:08.323962 15059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36219
I20260319 16:38:08.323994 15126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36219 every 8 connection(s)
I20260319 16:38:08.324271 15059 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:08.324801 15127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:08.325896 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15059
I20260319 16:38:08.325995 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:08.326153 25327 external_mini_cluster.cc:1468] Setting key 76858871eca31d2bf27dfb219fe9cef4
I20260319 16:38:08.327427 15127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d: Bootstrap starting.
I20260319 16:38:08.328058 15127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:08.328316 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:08.326610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54713 (local address 127.0.0.1:36219)
0319 16:38:08.326743 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:08.326751 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:08.326785 (+ 34us) server_negotiation.cc:408] Connection header received
0319 16:38:08.327224 (+ 439us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:08.327230 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:08.327253 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:08.327371 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:08.327988 (+ 617us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:08.327991 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:08.327991 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:08.327995 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:08.328019 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:08.328025 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:08.328091 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:08.328168 (+ 77us) server_negotiation.cc:300] Negotiation successful
0319 16:38:08.328201 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260319 16:38:08.328331 15127 log.cc:826] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:08.328949 15127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d: No bootstrap required, opened a new log
I20260319 16:38:08.329174 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:08.330111 15127 raft_consensus.cc:359] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304dca8e4ff64d0796f0b35878f5722d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260319 16:38:08.330242 15127 raft_consensus.cc:385] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:08.330272 15127 raft_consensus.cc:740] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304dca8e4ff64d0796f0b35878f5722d, State: Initialized, Role: FOLLOWER
I20260319 16:38:08.330365 15127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304dca8e4ff64d0796f0b35878f5722d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260319 16:38:08.330430 15127 raft_consensus.cc:399] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:08.330457 15127 raft_consensus.cc:493] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:08.330498 15127 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:08.331035 15127 raft_consensus.cc:515] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304dca8e4ff64d0796f0b35878f5722d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260319 16:38:08.331133 15127 leader_election.cc:304] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304dca8e4ff64d0796f0b35878f5722d; no voters:
I20260319 16:38:08.331267 15127 leader_election.cc:290] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:08.331401 15137 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:08.331489 15127 sys_catalog.cc:565] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:08.331552 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:08.330589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54725 (local address 127.0.0.1:36219)
0319 16:38:08.330638 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:08.330640 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:08.330648 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:38:08.330686 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:08.330687 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:08.330691 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:08.330735 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:08.331251 (+ 516us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:08.331252 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:08.331253 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:08.331257 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:08.331272 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:08.331276 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:08.331342 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:08.331437 (+ 95us) server_negotiation.cc:300] Negotiation successful
0319 16:38:08.331451 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:38:08.331647 15137 raft_consensus.cc:697] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [term 1 LEADER]: Becoming Leader. State: Replica: 304dca8e4ff64d0796f0b35878f5722d, State: Running, Role: LEADER
I20260319 16:38:08.331727 15137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304dca8e4ff64d0796f0b35878f5722d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } }
I20260319 16:38:08.332015 15137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304dca8e4ff64d0796f0b35878f5722d. Latest consensus state: current_term: 1 leader_uuid: "304dca8e4ff64d0796f0b35878f5722d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304dca8e4ff64d0796f0b35878f5722d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } } }
I20260319 16:38:08.332088 15137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:08.332230 15138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "304dca8e4ff64d0796f0b35878f5722d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304dca8e4ff64d0796f0b35878f5722d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36219 } } }
I20260319 16:38:08.332276 15145 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:08.332348 15138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:08.332798 15145 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:08.334239 15145 catalog_manager.cc:1357] Generated new cluster ID: 6c534af27239403989efc7cc2487d981
I20260319 16:38:08.334281 15145 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:08.356648 15145 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:08.356809 15145 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:08.365072 15145 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 304dca8e4ff64d0796f0b35878f5722d: Generated new TSK 0
I20260319 16:38:08.365298 15145 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:08.371654 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15059
2026-03-19T16:38:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2121 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:08 dist-test-slave-svn8 krb5kdc[15158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:08 dist-test-slave-svn8 krb5kdc[15158](info): set up 2 sockets
Mar 19 16:38:08 dist-test-slave-svn8 krb5kdc[15158](info): commencing operation
krb5kdc: starting...
W20260319 16:38:10.218634 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.829s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:10 dist-test-slave-svn8 krb5kdc[15158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:10.361279 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:46589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:10.435560 15174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:10.435741 15174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:10.435771 15174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:10.437433 15174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:10.437484 15174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:10.437503 15174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:10.437520 15174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:10.437533 15174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:10.439217 15174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:10.439456 15174 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:10.439620 15174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:10.441931 15179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:10.441931 15182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:10.442047 15180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:10.442327 15174 server_base.cc:1060] running on GCE node
I20260319 16:38:10.442549 15174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:10.442780 15174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:10.444000 15174 hybrid_clock.cc:648] HybridClock initialized: now 1773938290443978 us; error 32 us; skew 500 ppm
Mar 19 16:38:10 dist-test-slave-svn8 krb5kdc[15158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:10.446470 15174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:10.446998 15174 webserver.cc:492] Webserver started at http://127.0.0.1:34433/ using document root <none> and password file <none>
I20260319 16:38:10.447166 15174 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:10.447201 15174 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:10.447310 15174 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:10.448228 15174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "fff449a0c5c8414cb39b37d4ba26caea"
format_stamp: "Formatted at 2026-03-19 16:38:10 on dist-test-slave-svn8"
server_key: "ba3a8903b51a6c197e47713834f56df4"
server_key_iv: "35ded08734d594541227fa391942b6a7"
server_key_version: "encryptionkey@0"
I20260319 16:38:10.448542 15174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "fff449a0c5c8414cb39b37d4ba26caea"
format_stamp: "Formatted at 2026-03-19 16:38:10 on dist-test-slave-svn8"
server_key: "ba3a8903b51a6c197e47713834f56df4"
server_key_iv: "35ded08734d594541227fa391942b6a7"
server_key_version: "encryptionkey@0"
I20260319 16:38:10.449748 15174 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:38:10.450508 15189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:10.450721 15174 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:38:10.450784 15174 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "fff449a0c5c8414cb39b37d4ba26caea"
format_stamp: "Formatted at 2026-03-19 16:38:10 on dist-test-slave-svn8"
server_key: "ba3a8903b51a6c197e47713834f56df4"
server_key_iv: "35ded08734d594541227fa391942b6a7"
server_key_version: "encryptionkey@0"
I20260319 16:38:10.450850 15174 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:10.464051 15174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:10.465962 15174 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:10.466077 15174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:10.469935 15174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38055
I20260319 16:38:10.469969 15241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:38055 every 8 connection(s)
I20260319 16:38:10.470276 15174 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:10.470824 15242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:10.473133 15242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea: Bootstrap starting.
I20260319 16:38:10.473663 15242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:10.473923 15242 log.cc:826] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:10.474516 15242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea: No bootstrap required, opened a new log
I20260319 16:38:10.475683 15242 raft_consensus.cc:359] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff449a0c5c8414cb39b37d4ba26caea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38055 } }
I20260319 16:38:10.475798 15242 raft_consensus.cc:385] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:10.475816 15242 raft_consensus.cc:740] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff449a0c5c8414cb39b37d4ba26caea, State: Initialized, Role: FOLLOWER
I20260319 16:38:10.475946 15242 consensus_queue.cc:260] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff449a0c5c8414cb39b37d4ba26caea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38055 } }
I20260319 16:38:10.476017 15242 raft_consensus.cc:399] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:10.476055 15242 raft_consensus.cc:493] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:10.476065 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15174
I20260319 16:38:10.476096 15242 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:10.476176 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:10.476367 25327 external_mini_cluster.cc:1468] Setting key 9010a3299f304633546d5b121edf47de
I20260319 16:38:10.476734 15242 raft_consensus.cc:515] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff449a0c5c8414cb39b37d4ba26caea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38055 } }
I20260319 16:38:10.476915 15242 leader_election.cc:304] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fff449a0c5c8414cb39b37d4ba26caea; no voters:
I20260319 16:38:10.477069 15242 leader_election.cc:290] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:10.477121 15247 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:10.477298 15242 sys_catalog.cc:565] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:10.477311 15247 raft_consensus.cc:697] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [term 1 LEADER]: Becoming Leader. State: Replica: fff449a0c5c8414cb39b37d4ba26caea, State: Running, Role: LEADER
I20260319 16:38:10.478219 15247 consensus_queue.cc:237] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff449a0c5c8414cb39b37d4ba26caea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38055 } }
I20260319 16:38:10.478765 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [sys.catalog]: SysCatalogTable state changed. Reason: New leader fff449a0c5c8414cb39b37d4ba26caea. Latest consensus state: current_term: 1 leader_uuid: "fff449a0c5c8414cb39b37d4ba26caea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff449a0c5c8414cb39b37d4ba26caea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38055 } } }
I20260319 16:38:10.478929 15248 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fff449a0c5c8414cb39b37d4ba26caea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff449a0c5c8414cb39b37d4ba26caea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38055 } } }
I20260319 16:38:10.479233 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:10.479238 15248 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea [sys.catalog]: This master's current role is: LEADER
W20260319 16:38:10.479570 15261 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:10.479609 15261 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:10.479722 15264 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:10.479866 15264 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:10.481354 15264 catalog_manager.cc:1357] Generated new cluster ID: 58799e97af6b4236bd42a30d687b76ef
I20260319 16:38:10.481400 15264 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Mar 19 16:38:10 dist-test-slave-svn8 krb5kdc[15158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260319 16:38:10.486125 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:10.476792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:33985 (local address 127.0.0.1:38055)
0319 16:38:10.476935 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:10.476943 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:10.476962 (+ 19us) server_negotiation.cc:408] Connection header received
0319 16:38:10.477088 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:10.477094 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:10.477190 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:10.477409 (+ 219us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:10.478166 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:10.479398 (+ 1232us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:10.480203 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:10.480451 (+ 248us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:10.482850 (+ 2399us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:10.482856 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:10.482856 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:10.482874 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:10.484749 (+ 1875us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:10.485131 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:10.485134 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:10.485135 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:10.485191 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:10.485368 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:10.485369 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:10.485369 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:10.485593 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:10.485773 (+ 180us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:10.485863 (+ 90us) server_negotiation.cc:300] Negotiation successful
0319 16:38:10.485965 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260319 16:38:10.487243 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:10.494926 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:10.488594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:47393 (local address 127.0.0.1:38055)
0319 16:38:10.488633 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:10.488635 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:10.488834 (+ 199us) server_negotiation.cc:408] Connection header received
0319 16:38:10.488877 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:10.488878 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:10.488888 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:10.488953 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:10.489670 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:10.490461 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:10.491112 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:10.491301 (+ 189us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:10.492366 (+ 1065us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:10.492369 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:10.492370 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:10.492376 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:10.493875 (+ 1499us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:10.494236 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:10.494237 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:10.494237 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:10.494267 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:10.494431 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:10.494431 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:10.494432 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:10.494533 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:10.494625 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:10.494767 (+ 142us) server_negotiation.cc:300] Negotiation successful
0319 16:38:10.494830 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:38:10.518981 15264 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:10.519544 15264 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:10.529273 15264 catalog_manager.cc:6044] T 00000000000000000000000000000000 P fff449a0c5c8414cb39b37d4ba26caea: Generated new TSK 0
I20260319 16:38:10.529454 15264 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:10.531749 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15174
W20260319 16:38:10.536335 15156 connection.cc:537] client connection to 127.0.0.1:38055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38055): Connection reset by peer (error 104)
2026-03-19T16:38:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2160 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:10 dist-test-slave-svn8 krb5kdc[15273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:10 dist-test-slave-svn8 krb5kdc[15273](info): set up 2 sockets
Mar 19 16:38:10 dist-test-slave-svn8 krb5kdc[15273](info): commencing operation
krb5kdc: starting...
W20260319 16:38:12.370301 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.820s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:12 dist-test-slave-svn8 krb5kdc[15273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:12.504206 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:12.577054 15289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:12.577221 15289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:12.577255 15289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:12.578742 15289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:12.578783 15289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:12.578802 15289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:12.578817 15289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:12.578832 15289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:12.580440 15289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:12.580639 15289 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:12.580794 15289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:12.582966 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:12.583005 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:12.583127 15289 server_base.cc:1060] running on GCE node
W20260319 16:38:12.583295 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:12.583629 15289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:12.583897 15289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:12.585075 15289 hybrid_clock.cc:648] HybridClock initialized: now 1773938292585053 us; error 37 us; skew 500 ppm
Mar 19 16:38:12 dist-test-slave-svn8 krb5kdc[15273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:12.587196 15289 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:12.587706 15289 webserver.cc:492] Webserver started at http://127.0.0.1:43987/ using document root <none> and password file <none>
I20260319 16:38:12.587862 15289 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:12.587895 15289 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:12.588008 15289 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:12.588847 15289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "b6437d4dd01146ea83f92eafc5997b07"
format_stamp: "Formatted at 2026-03-19 16:38:12 on dist-test-slave-svn8"
server_key: "b1a5793e0c69f61389097cb31b3e64d7"
server_key_iv: "d57dc8d90f3c86e8bf7d65bf8bdd6923"
server_key_version: "encryptionkey@0"
I20260319 16:38:12.589123 15289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "b6437d4dd01146ea83f92eafc5997b07"
format_stamp: "Formatted at 2026-03-19 16:38:12 on dist-test-slave-svn8"
server_key: "b1a5793e0c69f61389097cb31b3e64d7"
server_key_iv: "d57dc8d90f3c86e8bf7d65bf8bdd6923"
server_key_version: "encryptionkey@0"
I20260319 16:38:12.590185 15289 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:38:12.590806 15304 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:12.590961 15289 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:38:12.591022 15289 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "b6437d4dd01146ea83f92eafc5997b07"
format_stamp: "Formatted at 2026-03-19 16:38:12 on dist-test-slave-svn8"
server_key: "b1a5793e0c69f61389097cb31b3e64d7"
server_key_iv: "d57dc8d90f3c86e8bf7d65bf8bdd6923"
server_key_version: "encryptionkey@0"
I20260319 16:38:12.591081 15289 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:12.603868 15289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:12.605181 15289 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:12.605298 15289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:12.608856 15289 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38361
I20260319 16:38:12.608889 15356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:38361 every 8 connection(s)
I20260319 16:38:12.609141 15289 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:12.609647 15357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:12.611830 15357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07: Bootstrap starting.
I20260319 16:38:12.612361 15357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:12.612576 15357 log.cc:826] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:12.613149 15357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07: No bootstrap required, opened a new log
I20260319 16:38:12.614377 15357 raft_consensus.cc:359] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6437d4dd01146ea83f92eafc5997b07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260319 16:38:12.614488 15357 raft_consensus.cc:385] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:12.614506 15357 raft_consensus.cc:740] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6437d4dd01146ea83f92eafc5997b07, State: Initialized, Role: FOLLOWER
I20260319 16:38:12.614615 15357 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6437d4dd01146ea83f92eafc5997b07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260319 16:38:12.614672 15357 raft_consensus.cc:399] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:12.614692 15357 raft_consensus.cc:493] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:12.614734 15357 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:12.615219 15357 raft_consensus.cc:515] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6437d4dd01146ea83f92eafc5997b07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260319 16:38:12.615307 15357 leader_election.cc:304] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6437d4dd01146ea83f92eafc5997b07; no voters:
I20260319 16:38:12.615444 15357 leader_election.cc:290] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:12.615494 15360 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:12.615621 15360 raft_consensus.cc:697] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [term 1 LEADER]: Becoming Leader. State: Replica: b6437d4dd01146ea83f92eafc5997b07, State: Running, Role: LEADER
I20260319 16:38:12.615643 15357 sys_catalog.cc:565] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:12.615698 15360 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6437d4dd01146ea83f92eafc5997b07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } }
I20260319 16:38:12.616015 15362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6437d4dd01146ea83f92eafc5997b07. Latest consensus state: current_term: 1 leader_uuid: "b6437d4dd01146ea83f92eafc5997b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6437d4dd01146ea83f92eafc5997b07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } } }
I20260319 16:38:12.615993 15361 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6437d4dd01146ea83f92eafc5997b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6437d4dd01146ea83f92eafc5997b07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38361 } } }
I20260319 16:38:12.616073 15362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:12.616084 15361 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:12.616250 15366 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:12.616708 15366 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:12.618014 15366 catalog_manager.cc:1357] Generated new cluster ID: f1b220e4790f4399ab37e695de06aa05
I20260319 16:38:12.618054 15366 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:12.618471 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15289
I20260319 16:38:12.618602 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:12.618777 25327 external_mini_cluster.cc:1468] Setting key 9b8f53142643dc39a323569931144efd
I20260319 16:38:12.623515 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:12.619185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:60575 (local address 127.0.0.1:38361)
0319 16:38:12.619324 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:12.619334 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:12.619513 (+ 179us) server_negotiation.cc:408] Connection header received
0319 16:38:12.619641 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:12.619647 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:12.619689 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:12.619798 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:12.620201 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:12.621315 (+ 1114us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:12.622137 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:12.622646 (+ 509us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:12.622834 (+ 188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:12.622838 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:12.622839 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:12.623037 (+ 198us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:12.623058 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:12.623065 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:12.623145 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:12.623248 (+ 103us) server_negotiation.cc:300] Negotiation successful
0319 16:38:12.623375 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260319 16:38:12.624322 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:12.628773 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:12.625797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:37691 (local address 127.0.0.1:38361)
0319 16:38:12.625825 (+ 28us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:12.625827 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:12.625927 (+ 100us) server_negotiation.cc:408] Connection header received
0319 16:38:12.625980 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:12.625981 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:12.625990 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:12.626039 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:12.626354 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:12.627243 (+ 889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:12.628012 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:12.628419 (+ 407us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:12.628543 (+ 124us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:12.628545 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:12.628545 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:12.628549 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:12.628559 (+ 10us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:12.628562 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:12.628604 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:12.628684 (+ 80us) server_negotiation.cc:300] Negotiation successful
0319 16:38:12.628692 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260319 16:38:12.630177 15366 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:12.630856 15366 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:12.653470 15366 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b6437d4dd01146ea83f92eafc5997b07: Generated new TSK 0
I20260319 16:38:12.653684 15366 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:12.657145 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15289
W20260319 16:38:12.661593 15271 connection.cc:537] client connection to 127.0.0.1:38361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38361): Connection reset by peer (error 104)
2026-03-19T16:38:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2125 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:12 dist-test-slave-svn8 krb5kdc[15388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:12 dist-test-slave-svn8 krb5kdc[15388](info): set up 2 sockets
Mar 19 16:38:12 dist-test-slave-svn8 krb5kdc[15388](info): commencing operation
krb5kdc: starting...
W20260319 16:38:14.490309 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:14 dist-test-slave-svn8 krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:14.626786 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:14.699954 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:14.700129 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:14.700150 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:14.701622 15404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:14.701665 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:14.701679 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:14.701690 15404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:14.701701 15404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:14.703351 15404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:14.703547 15404 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:14.703704 15404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:14.705919 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:14.705963 15404 server_base.cc:1060] running on GCE node
W20260319 16:38:14.706055 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:14.705933 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:14.706370 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:14.706574 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:14.707703 15404 hybrid_clock.cc:648] HybridClock initialized: now 1773938294707680 us; error 40 us; skew 500 ppm
Mar 19 16:38:14 dist-test-slave-svn8 krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:14.709956 15404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:14.710445 15404 webserver.cc:492] Webserver started at http://127.0.0.1:45043/ using document root <none> and password file <none>
I20260319 16:38:14.710613 15404 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:14.710646 15404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:14.710744 15404 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:14.711617 15404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "0eb56d2b66cd46af9b08e3a308caf911"
format_stamp: "Formatted at 2026-03-19 16:38:14 on dist-test-slave-svn8"
server_key: "58c943c67894caee07e3c22dab62a0ab"
server_key_iv: "8d89a2f5b36daf3428265ea1a675e2d8"
server_key_version: "encryptionkey@0"
I20260319 16:38:14.711882 15404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "0eb56d2b66cd46af9b08e3a308caf911"
format_stamp: "Formatted at 2026-03-19 16:38:14 on dist-test-slave-svn8"
server_key: "58c943c67894caee07e3c22dab62a0ab"
server_key_iv: "8d89a2f5b36daf3428265ea1a675e2d8"
server_key_version: "encryptionkey@0"
I20260319 16:38:14.712944 15404 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:38:14.713598 15419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:14.713783 15404 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:14.713832 15404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "0eb56d2b66cd46af9b08e3a308caf911"
format_stamp: "Formatted at 2026-03-19 16:38:14 on dist-test-slave-svn8"
server_key: "58c943c67894caee07e3c22dab62a0ab"
server_key_iv: "8d89a2f5b36daf3428265ea1a675e2d8"
server_key_version: "encryptionkey@0"
I20260319 16:38:14.713896 15404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:14.716934 15404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:14.723798 15404 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:14.723918 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:14.727605 15404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32849
I20260319 16:38:14.727664 15471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:32849 every 8 connection(s)
I20260319 16:38:14.727909 15404 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:14.728430 15472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:14.730871 15472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911: Bootstrap starting.
I20260319 16:38:14.731102 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15404
I20260319 16:38:14.731201 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:14.731369 25327 external_mini_cluster.cc:1468] Setting key 72e369ec52bee0c42dc9e80781488a81
I20260319 16:38:14.731487 15472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:14.731781 15472 log.cc:826] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:14.732560 15472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911: No bootstrap required, opened a new log
I20260319 16:38:14.733620 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:14.731823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:42543 (local address 127.0.0.1:32849)
0319 16:38:14.731987 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:14.731998 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:14.732088 (+ 90us) server_negotiation.cc:408] Connection header received
0319 16:38:14.732595 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:14.732605 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:14.732636 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:14.732775 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:14.733297 (+ 522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:14.733300 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:14.733301 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:14.733305 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:14.733327 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:14.733334 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:14.733418 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:14.733447 (+ 29us) server_negotiation.cc:300] Negotiation successful
0319 16:38:14.733480 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260319 16:38:14.734005 15472 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb56d2b66cd46af9b08e3a308caf911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260319 16:38:14.734136 15472 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:14.734169 15472 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb56d2b66cd46af9b08e3a308caf911, State: Initialized, Role: FOLLOWER
I20260319 16:38:14.734259 15472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb56d2b66cd46af9b08e3a308caf911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260319 16:38:14.734324 15472 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:14.734354 15472 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:14.734398 15472 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:14.734490 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:14.734966 15472 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb56d2b66cd46af9b08e3a308caf911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260319 16:38:14.735065 15472 leader_election.cc:304] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb56d2b66cd46af9b08e3a308caf911; no voters:
I20260319 16:38:14.735215 15472 leader_election.cc:290] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:14.735272 15477 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:14.735374 15477 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb56d2b66cd46af9b08e3a308caf911, State: Running, Role: LEADER
I20260319 16:38:14.735440 15472 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:14.735447 15477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb56d2b66cd46af9b08e3a308caf911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260319 16:38:14.735905 15481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb56d2b66cd46af9b08e3a308caf911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb56d2b66cd46af9b08e3a308caf911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260319 16:38:14.735973 15481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:14.736161 15479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb56d2b66cd46af9b08e3a308caf911. Latest consensus state: current_term: 1 leader_uuid: "0eb56d2b66cd46af9b08e3a308caf911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb56d2b66cd46af9b08e3a308caf911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260319 16:38:14.736269 15479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:14.736531 15487 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:14.737058 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:14.735880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:56409 (local address 127.0.0.1:32849)
0319 16:38:14.735916 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:14.735918 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:14.736035 (+ 117us) server_negotiation.cc:408] Connection header received
0319 16:38:14.736137 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:14.736138 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:14.736143 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:14.736170 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:14.736845 (+ 675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:14.736847 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:14.736847 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:14.736851 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:14.736868 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:14.736872 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:14.736922 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:14.736987 (+ 65us) server_negotiation.cc:300] Negotiation successful
0319 16:38:14.737001 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:38:14.737166 15487 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:14.738638 15487 catalog_manager.cc:1357] Generated new cluster ID: 280c531e663d49cfab4d14a151a75842
I20260319 16:38:14.738693 15487 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:14.748066 15487 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:14.748237 15487 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:14.754958 15487 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0eb56d2b66cd46af9b08e3a308caf911: Generated new TSK 0
I20260319 16:38:14.755124 15487 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:14.761845 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15404
2026-03-19T16:38:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2104 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:14 dist-test-slave-svn8 krb5kdc[15503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:14 dist-test-slave-svn8 krb5kdc[15503](info): set up 2 sockets
Mar 19 16:38:14 dist-test-slave-svn8 krb5kdc[15503](info): commencing operation
krb5kdc: starting...
W20260319 16:38:16.610704 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.831s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:16 dist-test-slave-svn8 krb5kdc[15503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:16.744671 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:16.821036 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:16.821215 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:16.821237 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:16.822821 15519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:16.822870 15519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:16.822885 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:16.822896 15519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:16.822906 15519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:16.824587 15519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:16.824793 15519 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:16.824954 15519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:16.827229 15527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:16.827255 15524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:16.827405 15525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:16.827493 15519 server_base.cc:1060] running on GCE node
I20260319 16:38:16.827678 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:16.827867 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:16.828997 15519 hybrid_clock.cc:648] HybridClock initialized: now 1773938296828974 us; error 35 us; skew 500 ppm
Mar 19 16:38:16 dist-test-slave-svn8 krb5kdc[15503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:16.831390 15519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:16.831873 15519 webserver.cc:492] Webserver started at http://127.0.0.1:39823/ using document root <none> and password file <none>
I20260319 16:38:16.832029 15519 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:16.832062 15519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:16.832158 15519 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:16.833027 15519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "d71a09ecc4744a229112c651960a5d24"
format_stamp: "Formatted at 2026-03-19 16:38:16 on dist-test-slave-svn8"
server_key: "67dd732efbe4e684b80270142706a7e8"
server_key_iv: "8078cc9e390db5bfa5716bd347aa5a6a"
server_key_version: "encryptionkey@0"
I20260319 16:38:16.833336 15519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "d71a09ecc4744a229112c651960a5d24"
format_stamp: "Formatted at 2026-03-19 16:38:16 on dist-test-slave-svn8"
server_key: "67dd732efbe4e684b80270142706a7e8"
server_key_iv: "8078cc9e390db5bfa5716bd347aa5a6a"
server_key_version: "encryptionkey@0"
I20260319 16:38:16.834471 15519 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:38:16.835064 15534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:16.835213 15519 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:38:16.835268 15519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "d71a09ecc4744a229112c651960a5d24"
format_stamp: "Formatted at 2026-03-19 16:38:16 on dist-test-slave-svn8"
server_key: "67dd732efbe4e684b80270142706a7e8"
server_key_iv: "8078cc9e390db5bfa5716bd347aa5a6a"
server_key_version: "encryptionkey@0"
I20260319 16:38:16.835325 15519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:16.850561 15519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:16.852182 15519 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:16.852313 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:16.856292 15519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32997
I20260319 16:38:16.856326 15586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:32997 every 8 connection(s)
I20260319 16:38:16.856628 15519 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:16.857151 15587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:16.859036 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15519
I20260319 16:38:16.859165 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:16.859328 25327 external_mini_cluster.cc:1468] Setting key 4df75904d1ceccae92285a3e0d2c8dc2
I20260319 16:38:16.859747 15587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24: Bootstrap starting.
I20260319 16:38:16.860381 15587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:16.860616 15587 log.cc:826] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:16.861213 15587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24: No bootstrap required, opened a new log
I20260319 16:38:16.862277 15587 raft_consensus.cc:359] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71a09ecc4744a229112c651960a5d24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260319 16:38:16.862396 15587 raft_consensus.cc:385] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:16.862414 15587 raft_consensus.cc:740] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71a09ecc4744a229112c651960a5d24, State: Initialized, Role: FOLLOWER
I20260319 16:38:16.862499 15587 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71a09ecc4744a229112c651960a5d24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260319 16:38:16.862552 15587 raft_consensus.cc:399] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:16.862586 15587 raft_consensus.cc:493] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:16.862627 15587 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:16.863143 15587 raft_consensus.cc:515] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71a09ecc4744a229112c651960a5d24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260319 16:38:16.863252 15587 leader_election.cc:304] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71a09ecc4744a229112c651960a5d24; no voters:
I20260319 16:38:16.863411 15587 leader_election.cc:290] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:16.863462 15592 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:16.863621 15592 raft_consensus.cc:697] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [term 1 LEADER]: Becoming Leader. State: Replica: d71a09ecc4744a229112c651960a5d24, State: Running, Role: LEADER
I20260319 16:38:16.863651 15587 sys_catalog.cc:565] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:16.864318 15592 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71a09ecc4744a229112c651960a5d24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
Mar 19 16:38:16 dist-test-slave-svn8 krb5kdc[15503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260319 16:38:16.864832 15593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71a09ecc4744a229112c651960a5d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71a09ecc4744a229112c651960a5d24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260319 16:38:16.864898 15593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:16.865096 15594 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71a09ecc4744a229112c651960a5d24. Latest consensus state: current_term: 1 leader_uuid: "d71a09ecc4744a229112c651960a5d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71a09ecc4744a229112c651960a5d24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260319 16:38:16.865141 15594 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24 [sys.catalog]: This master's current role is: LEADER
W20260319 16:38:16.865873 15607 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:16.865957 15607 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:16.866071 15603 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:16.866231 15603 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:16.867887 15603 catalog_manager.cc:1357] Generated new cluster ID: c17b7d98eed440d28fdc9b34cd594b70
I20260319 16:38:16.867935 15603 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:16.869297 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:16.859831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:34231 (local address 127.0.0.1:32997)
0319 16:38:16.859983 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:16.859992 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:16.860010 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:38:16.860349 (+ 339us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:16.860356 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:16.860394 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:16.860506 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:16.861359 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:16.862146 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:16.862794 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:16.863144 (+ 350us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:16.865495 (+ 2351us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:16.865504 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:16.865504 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:16.865525 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:16.867539 (+ 2014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:16.867995 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:16.867999 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:16.867999 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:16.868056 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:16.868257 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:16.868258 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:16.868258 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:16.868509 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:16.868649 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:16.869037 (+ 388us) server_negotiation.cc:300] Negotiation successful
0319 16:38:16.869128 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260319 16:38:16.870317 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:16.875465 15603 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:16.876101 15603 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:16.878476 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:16.871734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:53063 (local address 127.0.0.1:32997)
0319 16:38:16.871766 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:16.871768 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:16.871883 (+ 115us) server_negotiation.cc:408] Connection header received
0319 16:38:16.871930 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:16.871932 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:16.871943 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:16.871994 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:16.872785 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:16.873938 (+ 1153us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:16.874663 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:16.874821 (+ 158us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:16.875870 (+ 1049us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:16.875872 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:16.875873 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:16.875879 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:16.877410 (+ 1531us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:16.877828 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:16.877829 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:16.877829 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:16.877851 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:16.878017 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:16.878018 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:16.878018 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:16.878107 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:16.878192 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:16.878331 (+ 139us) server_negotiation.cc:300] Negotiation successful
0319 16:38:16.878386 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260319 16:38:16.884312 15603 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d71a09ecc4744a229112c651960a5d24: Generated new TSK 0
I20260319 16:38:16.884478 15603 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:16.893600 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15519
W20260319 16:38:16.897768 15501 connection.cc:537] client connection to 127.0.0.1:32997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32997): Connection reset by peer (error 104)
2026-03-19T16:38:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:16 dist-test-slave-svn8 krb5kdc[15618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:16 dist-test-slave-svn8 krb5kdc[15618](info): set up 2 sockets
Mar 19 16:38:16 dist-test-slave-svn8 krb5kdc[15618](info): commencing operation
krb5kdc: starting...
W20260319 16:38:18.742655 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.831s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:18 dist-test-slave-svn8 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:18.877557 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:36243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:18.950749 15634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:18.950924 15634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:18.950946 15634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:18.952380 15634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:18.952420 15634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:18.952433 15634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:18.952443 15634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:18.952454 15634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:18.954093 15634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:18.954257 15634 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:18.954381 15634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:18.956856 15640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:18.956820 15634 server_base.cc:1060] running on GCE node
W20260319 16:38:18.956804 15639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:18.956804 15642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:18.957234 15634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:18.957453 15634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:18.958611 15634 hybrid_clock.cc:648] HybridClock initialized: now 1773938298958589 us; error 32 us; skew 500 ppm
Mar 19 16:38:18 dist-test-slave-svn8 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:18.960958 15634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:18.961505 15634 webserver.cc:492] Webserver started at http://127.0.0.1:39269/ using document root <none> and password file <none>
I20260319 16:38:18.961668 15634 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:18.961702 15634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:18.961822 15634 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:18.962719 15634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "3523d7a5df1a445d89d07327f168c8fc"
format_stamp: "Formatted at 2026-03-19 16:38:18 on dist-test-slave-svn8"
server_key: "65fa4d2dee61255e490b4cd2eca342c9"
server_key_iv: "c5a82aa171d6041dbbfb9e66cb4a73a4"
server_key_version: "encryptionkey@0"
I20260319 16:38:18.962999 15634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "3523d7a5df1a445d89d07327f168c8fc"
format_stamp: "Formatted at 2026-03-19 16:38:18 on dist-test-slave-svn8"
server_key: "65fa4d2dee61255e490b4cd2eca342c9"
server_key_iv: "c5a82aa171d6041dbbfb9e66cb4a73a4"
server_key_version: "encryptionkey@0"
I20260319 16:38:18.964051 15634 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:38:18.964661 15649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:18.964813 15634 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:38:18.964869 15634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "3523d7a5df1a445d89d07327f168c8fc"
format_stamp: "Formatted at 2026-03-19 16:38:18 on dist-test-slave-svn8"
server_key: "65fa4d2dee61255e490b4cd2eca342c9"
server_key_iv: "c5a82aa171d6041dbbfb9e66cb4a73a4"
server_key_version: "encryptionkey@0"
I20260319 16:38:18.964927 15634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:18.977273 15634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:18.979050 15634 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:18.979177 15634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:18.983029 15634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42937
I20260319 16:38:18.983078 15701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42937 every 8 connection(s)
I20260319 16:38:18.983342 15634 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:18.983945 15702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:18.986428 15702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc: Bootstrap starting.
I20260319 16:38:18.986990 15702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:18.987202 15702 log.cc:826] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:18.987843 15702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc: No bootstrap required, opened a new log
I20260319 16:38:18.989018 15702 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3523d7a5df1a445d89d07327f168c8fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260319 16:38:18.989135 15702 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:18.989154 15702 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3523d7a5df1a445d89d07327f168c8fc, State: Initialized, Role: FOLLOWER
I20260319 16:38:18.989259 15702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3523d7a5df1a445d89d07327f168c8fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260319 16:38:18.989320 15702 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:18.989355 15702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:18.989394 15702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:18.989980 15702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3523d7a5df1a445d89d07327f168c8fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260319 16:38:18.990072 15702 leader_election.cc:304] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3523d7a5df1a445d89d07327f168c8fc; no voters:
I20260319 16:38:18.990206 15702 leader_election.cc:290] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:18.990279 15705 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:18.990410 15702 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:18.990440 15705 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [term 1 LEADER]: Becoming Leader. State: Replica: 3523d7a5df1a445d89d07327f168c8fc, State: Running, Role: LEADER
I20260319 16:38:18.990588 15705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3523d7a5df1a445d89d07327f168c8fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } }
I20260319 16:38:18.990945 15706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3523d7a5df1a445d89d07327f168c8fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3523d7a5df1a445d89d07327f168c8fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } } }
I20260319 16:38:18.990922 15707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3523d7a5df1a445d89d07327f168c8fc. Latest consensus state: current_term: 1 leader_uuid: "3523d7a5df1a445d89d07327f168c8fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3523d7a5df1a445d89d07327f168c8fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42937 } } }
I20260319 16:38:18.991065 15707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:18.991263 15714 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:18.991268 15706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:18.991751 15714 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:18.991917 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15634
I20260319 16:38:18.992022 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:18.992183 25327 external_mini_cluster.cc:1468] Setting key 4fd06707c44b0f74632166f8c68968e3
I20260319 16:38:18.993152 15714 catalog_manager.cc:1357] Generated new cluster ID: 6e2acbe9e65a41e3be30f85f0454bbee
I20260319 16:38:18.993227 15714 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:18.997222 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:18.992652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:42769 (local address 127.0.0.1:42937)
0319 16:38:18.992900 (+ 248us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:18.992910 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:18.993078 (+ 168us) server_negotiation.cc:408] Connection header received
0319 16:38:18.993278 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:18.993285 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:18.993332 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:18.993456 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:18.993874 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:18.995051 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:18.995780 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:18.996360 (+ 580us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:18.996582 (+ 222us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:18.996613 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:18.996613 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:18.996827 (+ 214us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:18.996849 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:18.996856 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:18.996952 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:18.997043 (+ 91us) server_negotiation.cc:300] Negotiation successful
0319 16:38:18.997068 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
I20260319 16:38:18.998432 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:19.003479 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:18.999826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:35913 (local address 127.0.0.1:42937)
0319 16:38:18.999864 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:18.999866 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:19.000014 (+ 148us) server_negotiation.cc:408] Connection header received
0319 16:38:19.000106 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:19.000107 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:19.000118 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:19.000145 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:19.000586 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:19.001261 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:19.002131 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:19.002308 (+ 177us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:19.003100 (+ 792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:19.003103 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:19.003104 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:19.003109 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:19.003125 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:19.003129 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:19.003166 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:19.003364 (+ 198us) server_negotiation.cc:300] Negotiation successful
0319 16:38:19.003379 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:38:19.012748 15714 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:19.013273 15714 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:19.025885 15714 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 3523d7a5df1a445d89d07327f168c8fc: Generated new TSK 0
I20260319 16:38:19.026062 15714 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:19.029351 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15634
W20260319 16:38:19.033448 15616 connection.cc:537] client connection to 127.0.0.1:42937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42937): Connection reset by peer (error 104)
2026-03-19T16:38:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2135 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:19 dist-test-slave-svn8 krb5kdc[15733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:19 dist-test-slave-svn8 krb5kdc[15733](info): set up 2 sockets
Mar 19 16:38:19 dist-test-slave-svn8 krb5kdc[15733](info): commencing operation
krb5kdc: starting...
W20260319 16:38:20.871078 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.824s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:20 dist-test-slave-svn8 krb5kdc[15733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:21.008054 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:21.081971 15749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:21.082151 15749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:21.082175 15749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:21.083753 15749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:21.083803 15749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:21.083817 15749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:21.083827 15749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:21.083837 15749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:21.085536 15749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:21.085773 15749 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:21.085943 15749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:21.088297 15754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:21.088297 15755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:21.088469 15749 server_base.cc:1060] running on GCE node
W20260319 16:38:21.088326 15757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:21.088732 15749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:21.088927 15749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:21.090046 15749 hybrid_clock.cc:648] HybridClock initialized: now 1773938301090034 us; error 28 us; skew 500 ppm
Mar 19 16:38:21 dist-test-slave-svn8 krb5kdc[15733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:21.092280 15749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260319 16:38:21.092767 15749 webserver.cc:492] Webserver started at http://127.0.0.1:45875/ using document root <none> and password file <none>
I20260319 16:38:21.092914 15749 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:21.092943 15749 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:21.093016 15749 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:21.093952 15749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "f197299103a24a19bab59638f4784d08"
format_stamp: "Formatted at 2026-03-19 16:38:21 on dist-test-slave-svn8"
server_key: "4a5ebd8d034b26910a187c6fa223f856"
server_key_iv: "eeb0e9b8b8fe34e21af8ea04418d776b"
server_key_version: "encryptionkey@0"
I20260319 16:38:21.094197 15749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "f197299103a24a19bab59638f4784d08"
format_stamp: "Formatted at 2026-03-19 16:38:21 on dist-test-slave-svn8"
server_key: "4a5ebd8d034b26910a187c6fa223f856"
server_key_iv: "eeb0e9b8b8fe34e21af8ea04418d776b"
server_key_version: "encryptionkey@0"
I20260319 16:38:21.095198 15749 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260319 16:38:21.095800 15764 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:21.095945 15749 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:38:21.095996 15749 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "f197299103a24a19bab59638f4784d08"
format_stamp: "Formatted at 2026-03-19 16:38:21 on dist-test-slave-svn8"
server_key: "4a5ebd8d034b26910a187c6fa223f856"
server_key_iv: "eeb0e9b8b8fe34e21af8ea04418d776b"
server_key_version: "encryptionkey@0"
I20260319 16:38:21.096055 15749 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:21.099076 15749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:21.104475 15749 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:21.104584 15749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:21.108292 15749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44121
I20260319 16:38:21.108323 15816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:44121 every 8 connection(s)
I20260319 16:38:21.108572 15749 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:21.109040 15817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:21.111358 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08: Bootstrap starting.
I20260319 16:38:21.111898 15817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:21.112051 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15749
I20260319 16:38:21.112166 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:21.112165 15817 log.cc:826] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:21.112339 25327 external_mini_cluster.cc:1468] Setting key 607497a729610cbb203256458809d27c
I20260319 16:38:21.112877 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08: No bootstrap required, opened a new log
I20260319 16:38:21.114154 15817 raft_consensus.cc:359] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f197299103a24a19bab59638f4784d08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44121 } }
I20260319 16:38:21.114261 15817 raft_consensus.cc:385] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:21.114288 15817 raft_consensus.cc:740] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f197299103a24a19bab59638f4784d08, State: Initialized, Role: FOLLOWER
I20260319 16:38:21.114392 15817 consensus_queue.cc:260] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f197299103a24a19bab59638f4784d08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44121 } }
I20260319 16:38:21.114445 15817 raft_consensus.cc:399] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:21.114473 15817 raft_consensus.cc:493] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:21.114758 15817 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:21.115191 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:21.112817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:49783 (local address 127.0.0.1:44121)
0319 16:38:21.112999 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:21.113009 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:21.113028 (+ 19us) server_negotiation.cc:408] Connection header received
0319 16:38:21.114135 (+ 1107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:21.114143 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:21.114170 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:21.114282 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:21.114888 (+ 606us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:21.114891 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:21.114892 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:21.114895 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:21.114914 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:21.114919 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:21.114981 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:21.115043 (+ 62us) server_negotiation.cc:300] Negotiation successful
0319 16:38:21.115075 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260319 16:38:21.115339 15817 raft_consensus.cc:515] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f197299103a24a19bab59638f4784d08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44121 } }
I20260319 16:38:21.115442 15817 leader_election.cc:304] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f197299103a24a19bab59638f4784d08; no voters:
I20260319 16:38:21.115586 15817 leader_election.cc:290] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:21.115636 15822 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:21.115773 15822 raft_consensus.cc:697] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [term 1 LEADER]: Becoming Leader. State: Replica: f197299103a24a19bab59638f4784d08, State: Running, Role: LEADER
I20260319 16:38:21.115778 15817 sys_catalog.cc:565] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:21.115875 15822 consensus_queue.cc:237] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f197299103a24a19bab59638f4784d08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44121 } }
I20260319 16:38:21.116048 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:21.117331 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f197299103a24a19bab59638f4784d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f197299103a24a19bab59638f4784d08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44121 } } }
I20260319 16:38:21.117419 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:21.117435 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f197299103a24a19bab59638f4784d08. Latest consensus state: current_term: 1 leader_uuid: "f197299103a24a19bab59638f4784d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f197299103a24a19bab59638f4784d08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44121 } } }
I20260319 16:38:21.117481 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08 [sys.catalog]: This master's current role is: LEADER
W20260319 16:38:21.117970 15841 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:21.118013 15841 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:21.118059 15842 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:21.118225 15842 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:21.118985 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:21.117763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:55695 (local address 127.0.0.1:44121)
0319 16:38:21.117792 (+ 29us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:21.117794 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:21.118032 (+ 238us) server_negotiation.cc:408] Connection header received
0319 16:38:21.118064 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:21.118065 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:21.118069 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:21.118129 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:21.118792 (+ 663us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:21.118793 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:21.118794 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:21.118797 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:21.118812 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:21.118816 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:21.118867 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:21.118932 (+ 65us) server_negotiation.cc:300] Negotiation successful
0319 16:38:21.118943 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260319 16:38:21.119937 15842 catalog_manager.cc:1357] Generated new cluster ID: b88df510700e480895caca2463984a3e
I20260319 16:38:21.119980 15842 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:21.126483 15842 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:21.126621 15842 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:21.131367 15842 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f197299103a24a19bab59638f4784d08: Generated new TSK 0
I20260319 16:38:21.131534 15842 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:21.133646 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15749
2026-03-19T16:38:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2104 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:21 dist-test-slave-svn8 krb5kdc[15848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:21 dist-test-slave-svn8 krb5kdc[15848](info): set up 2 sockets
Mar 19 16:38:21 dist-test-slave-svn8 krb5kdc[15848](info): commencing operation
krb5kdc: starting...
W20260319 16:38:22.966548 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:23 dist-test-slave-svn8 krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:23.101421 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36931
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:23.174888 15864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:23.175079 15864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:23.175109 15864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:23.176628 15864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:23.176680 15864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:23.176698 15864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:23.176714 15864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:23.176728 15864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:23.178395 15864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:36931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:23.178632 15864 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:23.178790 15864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:23.181066 15872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:23.181219 15869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:23.181248 15870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:23.181288 15864 server_base.cc:1060] running on GCE node
I20260319 16:38:23.181548 15864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:23.181821 15864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:23.182950 15864 hybrid_clock.cc:648] HybridClock initialized: now 1773938303182931 us; error 33 us; skew 500 ppm
Mar 19 16:38:23 dist-test-slave-svn8 krb5kdc[15848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938303, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:23.185119 15864 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:23.185608 15864 webserver.cc:492] Webserver started at http://127.24.187.254:41799/ using document root <none> and password file <none>
I20260319 16:38:23.185798 15864 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:23.185833 15864 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:23.185915 15864 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:23.186839 15864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "14bd7a037fc64128a92bf6e1d00d9d3e"
format_stamp: "Formatted at 2026-03-19 16:38:23 on dist-test-slave-svn8"
server_key: "2891afbd9f081c8ee533f220cbe795dd"
server_key_iv: "9ededfda48a67fe1e610b1fb367a323f"
server_key_version: "encryptionkey@0"
I20260319 16:38:23.187114 15864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "14bd7a037fc64128a92bf6e1d00d9d3e"
format_stamp: "Formatted at 2026-03-19 16:38:23 on dist-test-slave-svn8"
server_key: "2891afbd9f081c8ee533f220cbe795dd"
server_key_iv: "9ededfda48a67fe1e610b1fb367a323f"
server_key_version: "encryptionkey@0"
I20260319 16:38:23.188186 15864 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:38:23.188841 15879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:23.189011 15864 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:38:23.189067 15864 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "14bd7a037fc64128a92bf6e1d00d9d3e"
format_stamp: "Formatted at 2026-03-19 16:38:23 on dist-test-slave-svn8"
server_key: "2891afbd9f081c8ee533f220cbe795dd"
server_key_iv: "9ededfda48a67fe1e610b1fb367a323f"
server_key_version: "encryptionkey@0"
I20260319 16:38:23.189195 15864 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:23.204716 15864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:23.206518 15864 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:23.206617 15864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:23.210506 15864 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:36931
I20260319 16:38:23.210569 15931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:36931 every 8 connection(s)
I20260319 16:38:23.210821 15864 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:23.211369 15932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:23.213768 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e: Bootstrap starting.
I20260319 16:38:23.214334 15932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:23.214560 15932 log.cc:826] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:23.215219 15932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e: No bootstrap required, opened a new log
I20260319 16:38:23.215662 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15864
I20260319 16:38:23.215765 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:23.215924 25327 external_mini_cluster.cc:1468] Setting key 02bb8597b52236a4cf19d80ae1cdbff7
I20260319 16:38:23.216707 15932 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36931 } }
I20260319 16:38:23.216835 15932 raft_consensus.cc:385] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:23.216866 15932 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14bd7a037fc64128a92bf6e1d00d9d3e, State: Initialized, Role: FOLLOWER
I20260319 16:38:23.216988 15932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36931 } }
I20260319 16:38:23.217059 15932 raft_consensus.cc:399] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:23.217089 15932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:23.217133 15932 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:23.217826 15932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36931 } }
I20260319 16:38:23.217919 15932 leader_election.cc:304] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14bd7a037fc64128a92bf6e1d00d9d3e; no voters:
I20260319 16:38:23.218052 15932 leader_election.cc:290] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:23.218101 15937 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:23.218233 15937 raft_consensus.cc:697] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [term 1 LEADER]: Becoming Leader. State: Replica: 14bd7a037fc64128a92bf6e1d00d9d3e, State: Running, Role: LEADER
I20260319 16:38:23.218266 15932 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:23.218320 15937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36931 } }
I20260319 16:38:23.218606 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14bd7a037fc64128a92bf6e1d00d9d3e. Latest consensus state: current_term: 1 leader_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36931 } } }
I20260319 16:38:23.218616 15938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14bd7a037fc64128a92bf6e1d00d9d3e" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 36931 } } }
I20260319 16:38:23.218741 15938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:23.218917 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:23.219197 15943 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:23.220038 15943 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Mar 19 16:38:23 dist-test-slave-svn8 krb5kdc[15848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:38:23.221894 15943 catalog_manager.cc:1357] Generated new cluster ID: 6b2cd7d12af148d99961dc149968c6d9
I20260319 16:38:23.221938 15943 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:23.225538 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:23.216376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35791 (local address 127.24.187.254:36931)
0319 16:38:23.216512 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:23.216521 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:23.216539 (+ 18us) server_negotiation.cc:408] Connection header received
0319 16:38:23.216663 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:23.216667 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:23.216710 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:23.216803 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:23.217759 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:23.218855 (+ 1096us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:23.219818 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:23.221175 (+ 1357us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:23.222147 (+ 972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:23.222152 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:23.222153 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:23.222170 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:23.224028 (+ 1858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:23.224423 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:23.224425 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:23.224426 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:23.224487 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:23.224662 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:23.224663 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:23.224663 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:23.224873 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:23.224989 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:23.225157 (+ 168us) server_negotiation.cc:300] Negotiation successful
0319 16:38:23.225377 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260319 16:38:23.226615 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:23.234210 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:23.227851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50751 (local address 127.24.187.254:36931)
0319 16:38:23.227885 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:23.227887 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:23.228013 (+ 126us) server_negotiation.cc:408] Connection header received
0319 16:38:23.228044 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:23.228045 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:23.228056 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:23.228119 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:23.228824 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:23.229855 (+ 1031us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:23.230509 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:23.230983 (+ 474us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:23.231620 (+ 637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:23.231622 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:23.231623 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:23.231629 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:23.233071 (+ 1442us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:23.233433 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:23.233433 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:23.233434 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:23.233456 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:23.233616 (+ 160us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:23.233617 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:23.233617 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:23.233735 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:23.233832 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:23.234057 (+ 225us) server_negotiation.cc:300] Negotiation successful
0319 16:38:23.234118 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260319 16:38:23.265301 15943 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:23.265905 15943 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:23.271960 15943 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 14bd7a037fc64128a92bf6e1d00d9d3e: Generated new TSK 0
I20260319 16:38:23.272114 15943 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:23.283294 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15864
W20260319 16:38:23.287689 15846 connection.cc:537] client connection to 127.24.187.254:36931 recv error: Network error: failed to read from TLS socket (remote: 127.24.187.254:36931): Connection reset by peer (error 104)
2026-03-19T16:38:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2149 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:23 dist-test-slave-svn8 krb5kdc[15963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:23 dist-test-slave-svn8 krb5kdc[15963](info): set up 2 sockets
Mar 19 16:38:23 dist-test-slave-svn8 krb5kdc[15963](info): commencing operation
krb5kdc: starting...
W20260319 16:38:25.120007 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:25 dist-test-slave-svn8 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:25.252821 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33505
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:25.325866 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:25.326032 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:25.326054 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:25.327564 15979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:25.327615 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:25.327627 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:25.327639 15979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:25.327649 15979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:25.329272 15979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:33505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:25.329463 15979 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:25.329641 15979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:25.331852 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:25.331848 15984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:25.331933 15979 server_base.cc:1060] running on GCE node
W20260319 16:38:25.331848 15985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:25.332331 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:25.332531 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:25.333675 15979 hybrid_clock.cc:648] HybridClock initialized: now 1773938305333658 us; error 30 us; skew 500 ppm
Mar 19 16:38:25 dist-test-slave-svn8 krb5kdc[15963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938305, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:25.335835 15979 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:25.336309 15979 webserver.cc:492] Webserver started at http://127.24.187.254:41803/ using document root <none> and password file <none>
I20260319 16:38:25.336468 15979 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:25.336500 15979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:25.336597 15979 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:25.337466 15979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "0651fe428f864021ae643f9362977551"
format_stamp: "Formatted at 2026-03-19 16:38:25 on dist-test-slave-svn8"
server_key: "afb3021430bb5bfde682685adf2b11f2"
server_key_iv: "ac7d6eeb27ff58613b9255a2d74d171a"
server_key_version: "encryptionkey@0"
I20260319 16:38:25.337796 15979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "0651fe428f864021ae643f9362977551"
format_stamp: "Formatted at 2026-03-19 16:38:25 on dist-test-slave-svn8"
server_key: "afb3021430bb5bfde682685adf2b11f2"
server_key_iv: "ac7d6eeb27ff58613b9255a2d74d171a"
server_key_version: "encryptionkey@0"
I20260319 16:38:25.338847 15979 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:38:25.339501 15994 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:25.339684 15979 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:38:25.339745 15979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "0651fe428f864021ae643f9362977551"
format_stamp: "Formatted at 2026-03-19 16:38:25 on dist-test-slave-svn8"
server_key: "afb3021430bb5bfde682685adf2b11f2"
server_key_iv: "ac7d6eeb27ff58613b9255a2d74d171a"
server_key_version: "encryptionkey@0"
I20260319 16:38:25.339793 15979 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:25.347718 15979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:25.349345 15979 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:25.349436 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:25.353116 15979 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:33505
I20260319 16:38:25.353152 16046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:33505 every 8 connection(s)
I20260319 16:38:25.353495 15979 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:25.354058 16047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:25.356343 16047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551: Bootstrap starting.
I20260319 16:38:25.356879 16047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:25.356940 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 15979
I20260319 16:38:25.357069 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:25.357125 16047 log.cc:826] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:25.357224 25327 external_mini_cluster.cc:1468] Setting key 8599283e1a9171d7cca84270f5013bd8
I20260319 16:38:25.357864 16047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551: No bootstrap required, opened a new log
I20260319 16:38:25.359289 16047 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0651fe428f864021ae643f9362977551" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33505 } }
I20260319 16:38:25.359403 16047 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:25.359423 16047 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0651fe428f864021ae643f9362977551, State: Initialized, Role: FOLLOWER
I20260319 16:38:25.359509 16047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0651fe428f864021ae643f9362977551" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33505 } }
I20260319 16:38:25.359571 16047 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:25.359594 16047 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:25.359635 16047 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:25.360165 16047 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0651fe428f864021ae643f9362977551" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33505 } }
I20260319 16:38:25.360267 16047 leader_election.cc:304] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0651fe428f864021ae643f9362977551; no voters:
I20260319 16:38:25.360414 16047 leader_election.cc:290] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:25.360461 16052 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:25.360620 16047 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:25.360627 16052 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [term 1 LEADER]: Becoming Leader. State: Replica: 0651fe428f864021ae643f9362977551, State: Running, Role: LEADER
I20260319 16:38:25.360873 16052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0651fe428f864021ae643f9362977551" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33505 } }
I20260319 16:38:25.361174 16052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0651fe428f864021ae643f9362977551. Latest consensus state: current_term: 1 leader_uuid: "0651fe428f864021ae643f9362977551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0651fe428f864021ae643f9362977551" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33505 } } }
I20260319 16:38:25.361281 16052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:25.361506 16061 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:25.361549 16054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0651fe428f864021ae643f9362977551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0651fe428f864021ae643f9362977551" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33505 } } }
I20260319 16:38:25.361627 16054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:25.361941 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:25.357703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56709 (local address 127.24.187.254:33505)
0319 16:38:25.357868 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:25.357882 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:25.357902 (+ 20us) server_negotiation.cc:408] Connection header received
0319 16:38:25.358235 (+ 333us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:25.358241 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:25.358278 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:25.358377 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:25.358795 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:25.359596 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:25.360162 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:25.360418 (+ 256us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:25.360829 (+ 411us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:25.360836 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:25.360837 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:25.361028 (+ 191us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:25.361049 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:25.361057 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:25.361159 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:25.361773 (+ 614us) server_negotiation.cc:300] Negotiation successful
0319 16:38:25.361799 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260319 16:38:25.361986 16061 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:25.363018 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:25.363602 16061 catalog_manager.cc:1357] Generated new cluster ID: e7a6fd1368514e5ba833f0c812f0e147
I20260319 16:38:25.363638 16061 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:25.367272 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:25.364453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39407 (local address 127.24.187.254:33505)
0319 16:38:25.364499 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:25.364501 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:25.364643 (+ 142us) server_negotiation.cc:408] Connection header received
0319 16:38:25.364684 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:25.364685 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:25.364695 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:25.364763 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:25.365088 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:25.365797 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:25.366474 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:25.366942 (+ 468us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:25.367036 (+ 94us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:25.367039 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:25.367039 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:25.367043 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:25.367053 (+ 10us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:25.367057 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:25.367113 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:25.367184 (+ 71us) server_negotiation.cc:300] Negotiation successful
0319 16:38:25.367196 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:38:25.400588 16061 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:25.401337 16061 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:25.413358 16061 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0651fe428f864021ae643f9362977551: Generated new TSK 0
I20260319 16:38:25.413543 16061 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:25.424841 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 15979
W20260319 16:38:25.429088 15961 connection.cc:537] client connection to 127.24.187.254:33505 recv error: Network error: failed to read from TLS socket (remote: 127.24.187.254:33505): Connection reset by peer (error 104)
2026-03-19T16:38:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2141 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:25 dist-test-slave-svn8 krb5kdc[16078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:25 dist-test-slave-svn8 krb5kdc[16078](info): set up 2 sockets
Mar 19 16:38:25 dist-test-slave-svn8 krb5kdc[16078](info): commencing operation
krb5kdc: starting...
W20260319 16:38:27.262674 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.820s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:27 dist-test-slave-svn8 krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:27.405756 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39343
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:27.481032 16094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:27.481205 16094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:27.481226 16094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:27.482705 16094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:27.482745 16094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:27.482758 16094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:27.482769 16094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:27.482779 16094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:27.484340 16094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:39343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:27.484500 16094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:27.484623 16094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:27.487043 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:27.487066 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:27.487123 16094 server_base.cc:1060] running on GCE node
W20260319 16:38:27.487067 16099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:27.487419 16094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:27.487649 16094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:27.488809 16094 hybrid_clock.cc:648] HybridClock initialized: now 1773938307488785 us; error 24 us; skew 500 ppm
Mar 19 16:38:27 dist-test-slave-svn8 krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:27.491011 16094 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:27.491528 16094 webserver.cc:492] Webserver started at http://127.24.187.254:42223/ using document root <none> and password file <none>
I20260319 16:38:27.491688 16094 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:27.491724 16094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:27.491820 16094 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:27.492889 16094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "42e6f53be2ef4a04b3be732f9a98f5d8"
format_stamp: "Formatted at 2026-03-19 16:38:27 on dist-test-slave-svn8"
server_key: "9e091b12e64066e03b33ab1db75d7c9f"
server_key_iv: "fda481fb81dd28ec80ec7826642a547f"
server_key_version: "encryptionkey@0"
I20260319 16:38:27.493211 16094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "42e6f53be2ef4a04b3be732f9a98f5d8"
format_stamp: "Formatted at 2026-03-19 16:38:27 on dist-test-slave-svn8"
server_key: "9e091b12e64066e03b33ab1db75d7c9f"
server_key_iv: "fda481fb81dd28ec80ec7826642a547f"
server_key_version: "encryptionkey@0"
I20260319 16:38:27.494397 16094 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260319 16:38:27.495098 16109 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:27.495275 16094 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260319 16:38:27.495368 16094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "42e6f53be2ef4a04b3be732f9a98f5d8"
format_stamp: "Formatted at 2026-03-19 16:38:27 on dist-test-slave-svn8"
server_key: "9e091b12e64066e03b33ab1db75d7c9f"
server_key_iv: "fda481fb81dd28ec80ec7826642a547f"
server_key_version: "encryptionkey@0"
I20260319 16:38:27.495427 16094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:27.498461 16094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:27.504796 16094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:27.504906 16094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:27.508699 16094 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:39343
I20260319 16:38:27.508735 16161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:39343 every 8 connection(s)
I20260319 16:38:27.508998 16094 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:27.509513 16162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:27.510033 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16094
I20260319 16:38:27.510135 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:27.510304 25327 external_mini_cluster.cc:1468] Setting key b4233138cc6a4cca111981379d7756b5
I20260319 16:38:27.512121 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8: Bootstrap starting.
I20260319 16:38:27.512341 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:27.510765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58695 (local address 127.24.187.254:39343)
0319 16:38:27.510929 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:27.510939 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:27.510960 (+ 21us) server_negotiation.cc:408] Connection header received
0319 16:38:27.511263 (+ 303us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:27.511268 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:27.511297 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:27.511402 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:27.512003 (+ 601us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:27.512006 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:27.512006 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:27.512011 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:27.512028 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:27.512034 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:27.512112 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:27.512142 (+ 30us) server_negotiation.cc:300] Negotiation successful
0319 16:38:27.512172 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260319 16:38:27.512691 16162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:27.512900 16162 log.cc:826] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:27.513087 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:27.513471 16162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8: No bootstrap required, opened a new log
I20260319 16:38:27.514683 16162 raft_consensus.cc:359] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39343 } }
I20260319 16:38:27.514794 16162 raft_consensus.cc:385] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:27.514819 16162 raft_consensus.cc:740] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e6f53be2ef4a04b3be732f9a98f5d8, State: Initialized, Role: FOLLOWER
I20260319 16:38:27.514918 16162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39343 } }
I20260319 16:38:27.514984 16162 raft_consensus.cc:399] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:27.515013 16162 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:27.515051 16162 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:27.515632 16162 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39343 } }
I20260319 16:38:27.515633 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:27.514577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42457 (local address 127.24.187.254:39343)
0319 16:38:27.514624 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:27.514626 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:27.514679 (+ 53us) server_negotiation.cc:408] Connection header received
0319 16:38:27.514743 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:27.514744 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:27.514748 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:27.514793 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:27.515423 (+ 630us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:27.515425 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:27.515425 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:27.515429 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:27.515443 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:27.515447 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:27.515494 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:27.515542 (+ 48us) server_negotiation.cc:300] Negotiation successful
0319 16:38:27.515553 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:38:27.515765 16162 leader_election.cc:304] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42e6f53be2ef4a04b3be732f9a98f5d8; no voters:
I20260319 16:38:27.515998 16162 leader_election.cc:290] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:27.516100 16172 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:27.516234 16162 sys_catalog.cc:565] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:27.516369 16172 raft_consensus.cc:697] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 42e6f53be2ef4a04b3be732f9a98f5d8, State: Running, Role: LEADER
I20260319 16:38:27.516471 16172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39343 } }
I20260319 16:38:27.516815 16174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e6f53be2ef4a04b3be732f9a98f5d8. Latest consensus state: current_term: 1 leader_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39343 } } }
I20260319 16:38:27.516832 16173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e6f53be2ef4a04b3be732f9a98f5d8" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 39343 } } }
I20260319 16:38:27.516980 16174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:27.516990 16173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:27.517460 16181 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:27.517930 16181 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:27.519222 16181 catalog_manager.cc:1357] Generated new cluster ID: 3bf7f8c4b8f241d2aa5517bfa62170fe
I20260319 16:38:27.519269 16181 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:27.531880 16181 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:27.532016 16181 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:27.537411 16181 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 42e6f53be2ef4a04b3be732f9a98f5d8: Generated new TSK 0
I20260319 16:38:27.537606 16181 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:27.544349 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16094
2026-03-19T16:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2119 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:27 dist-test-slave-svn8 krb5kdc[16193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:27 dist-test-slave-svn8 krb5kdc[16193](info): set up 2 sockets
Mar 19 16:38:27 dist-test-slave-svn8 krb5kdc[16193](info): commencing operation
krb5kdc: starting...
W20260319 16:38:29.382541 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.820s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:29 dist-test-slave-svn8 krb5kdc[16193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:29.516825 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33417
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:33417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:29.590240 16209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:29.590425 16209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:29.590456 16209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:29.591952 16209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:29.591998 16209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:29.592017 16209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:29.592033 16209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:29.592048 16209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:29.593659 16209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:33417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:29.593878 16209 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:29.594029 16209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:29.596240 16215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:29.596331 16217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:29.596463 16214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:29.596366 16209 server_base.cc:1060] running on GCE node
I20260319 16:38:29.596832 16209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:29.597048 16209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:29.598214 16209 hybrid_clock.cc:648] HybridClock initialized: now 1773938309598187 us; error 37 us; skew 500 ppm
Mar 19 16:38:29 dist-test-slave-svn8 krb5kdc[16193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:29.600557 16209 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:29.601034 16209 webserver.cc:492] Webserver started at http://127.24.187.254:40063/ using document root <none> and password file <none>
I20260319 16:38:29.601176 16209 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:29.601204 16209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:29.601289 16209 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:29.602196 16209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "23373ffe62b845ff853b3760a5f46b70"
format_stamp: "Formatted at 2026-03-19 16:38:29 on dist-test-slave-svn8"
server_key: "a099a67079e39b40526c87c3ca0ba67c"
server_key_iv: "54aa8330cd2c83ffd7f3e979d68d70e0"
server_key_version: "encryptionkey@0"
I20260319 16:38:29.602478 16209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "23373ffe62b845ff853b3760a5f46b70"
format_stamp: "Formatted at 2026-03-19 16:38:29 on dist-test-slave-svn8"
server_key: "a099a67079e39b40526c87c3ca0ba67c"
server_key_iv: "54aa8330cd2c83ffd7f3e979d68d70e0"
server_key_version: "encryptionkey@0"
I20260319 16:38:29.603583 16209 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:38:29.604163 16224 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:29.604368 16209 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:38:29.604427 16209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "23373ffe62b845ff853b3760a5f46b70"
format_stamp: "Formatted at 2026-03-19 16:38:29 on dist-test-slave-svn8"
server_key: "a099a67079e39b40526c87c3ca0ba67c"
server_key_iv: "54aa8330cd2c83ffd7f3e979d68d70e0"
server_key_version: "encryptionkey@0"
I20260319 16:38:29.604489 16209 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:29.645229 16209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:29.647022 16209 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:29.647140 16209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:29.650978 16209 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:33417
I20260319 16:38:29.651018 16276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:33417 every 8 connection(s)
I20260319 16:38:29.651329 16209 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:29.651859 16277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:29.654240 16277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70: Bootstrap starting.
I20260319 16:38:29.654783 16277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:29.655009 16277 log.cc:826] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:29.655751 16277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70: No bootstrap required, opened a new log
I20260319 16:38:29.656934 16277 raft_consensus.cc:359] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23373ffe62b845ff853b3760a5f46b70" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33417 } }
I20260319 16:38:29.657047 16277 raft_consensus.cc:385] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:29.657065 16277 raft_consensus.cc:740] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23373ffe62b845ff853b3760a5f46b70, State: Initialized, Role: FOLLOWER
I20260319 16:38:29.657176 16277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23373ffe62b845ff853b3760a5f46b70" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33417 } }
I20260319 16:38:29.657231 16277 raft_consensus.cc:399] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:29.657250 16277 raft_consensus.cc:493] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:29.657276 16277 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:29.657845 16277 raft_consensus.cc:515] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23373ffe62b845ff853b3760a5f46b70" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33417 } }
I20260319 16:38:29.657943 16277 leader_election.cc:304] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23373ffe62b845ff853b3760a5f46b70; no voters:
I20260319 16:38:29.658075 16277 leader_election.cc:290] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:29.658133 16280 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:29.658291 16277 sys_catalog.cc:565] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:29.658329 16280 raft_consensus.cc:697] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [term 1 LEADER]: Becoming Leader. State: Replica: 23373ffe62b845ff853b3760a5f46b70, State: Running, Role: LEADER
I20260319 16:38:29.658450 16280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23373ffe62b845ff853b3760a5f46b70" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33417 } }
I20260319 16:38:29.658802 16282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23373ffe62b845ff853b3760a5f46b70. Latest consensus state: current_term: 1 leader_uuid: "23373ffe62b845ff853b3760a5f46b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23373ffe62b845ff853b3760a5f46b70" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33417 } } }
I20260319 16:38:29.658882 16282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:29.658972 16281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23373ffe62b845ff853b3760a5f46b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23373ffe62b845ff853b3760a5f46b70" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 33417 } } }
I20260319 16:38:29.659142 16281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:29.659255 16289 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:29.659921 16289 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:29.661299 16289 catalog_manager.cc:1357] Generated new cluster ID: 65e1f56607174d57875598344aadb45b
I20260319 16:38:29.661352 16289 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:29.665930 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16209
I20260319 16:38:29.666036 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:29.666191 25327 external_mini_cluster.cc:1468] Setting key 8ab38c5a53c9b16a7846ade9e0218c56
I20260319 16:38:29.668210 16289 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:29.668919 16289 catalog_manager.cc:1514] Loading token signing keys...
Mar 19 16:38:29 dist-test-slave-svn8 krb5kdc[16193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:38:29.677657 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:29.666614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55533 (local address 127.24.187.254:33417)
0319 16:38:29.666742 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:29.666750 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:29.669961 (+ 3211us) server_negotiation.cc:408] Connection header received
0319 16:38:29.670090 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:29.670095 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:29.670133 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:29.670285 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:29.671022 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:29.671821 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:29.672446 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:29.672683 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:29.674833 (+ 2150us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:29.674840 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:29.674840 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:29.674857 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:29.676459 (+ 1602us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:29.676764 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:29.676767 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:29.676767 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:29.676814 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:29.676953 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:29.676954 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:29.676955 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:29.677108 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:29.677248 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:29.677357 (+ 109us) server_negotiation.cc:300] Negotiation successful
0319 16:38:29.677476 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260319 16:38:29.678469 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:29.680200 16289 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 23373ffe62b845ff853b3760a5f46b70: Generated new TSK 0
I20260319 16:38:29.680365 16289 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:29.686893 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:29.679742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36991 (local address 127.24.187.254:33417)
0319 16:38:29.679775 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:29.679777 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:29.681942 (+ 2165us) server_negotiation.cc:408] Connection header received
0319 16:38:29.681997 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:29.681998 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:29.682020 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:29.682064 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:29.682666 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:29.683153 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:29.683697 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:29.683861 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:29.684759 (+ 898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:29.684762 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:29.684762 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:29.684769 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:29.686059 (+ 1290us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:29.686316 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:29.686317 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:29.686317 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:29.686337 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:29.686466 (+ 129us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:29.686467 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:29.686467 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:29.686547 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:29.686634 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:29.686724 (+ 90us) server_negotiation.cc:300] Negotiation successful
0319 16:38:29.686800 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260319 16:38:29.688678 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16209
2026-03-19T16:38:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:29 dist-test-slave-svn8 krb5kdc[16308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:29 dist-test-slave-svn8 krb5kdc[16308](info): set up 2 sockets
Mar 19 16:38:29 dist-test-slave-svn8 krb5kdc[16308](info): commencing operation
krb5kdc: starting...
W20260319 16:38:31.526316 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:31 dist-test-slave-svn8 krb5kdc[16308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:31.663038 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:46391
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:46391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:31.737454 16324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:31.737638 16324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:31.737660 16324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:31.739162 16324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:31.739209 16324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:31.739221 16324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:31.739233 16324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:31.739244 16324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:31.740839 16324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:46391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:46391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:31.741015 16324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:31.741144 16324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:31.743561 16330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:31.743667 16324 server_base.cc:1060] running on GCE node
W20260319 16:38:31.743641 16329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:31.743605 16332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:31.744117 16324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:31.744361 16324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:31.745525 16324 hybrid_clock.cc:648] HybridClock initialized: now 1773938311745510 us; error 29 us; skew 500 ppm
Mar 19 16:38:31 dist-test-slave-svn8 krb5kdc[16308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938311, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:31.747936 16324 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:31.748507 16324 webserver.cc:492] Webserver started at http://127.24.187.254:35247/ using document root <none> and password file <none>
I20260319 16:38:31.748679 16324 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:31.748714 16324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:31.748814 16324 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:31.749773 16324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "ae364743f5de4902864a43ec6f4fd96b"
format_stamp: "Formatted at 2026-03-19 16:38:31 on dist-test-slave-svn8"
server_key: "b0bb069b840f29affa21dea88ac880ec"
server_key_iv: "438b83f39404307962b0f5cd09fc969a"
server_key_version: "encryptionkey@0"
I20260319 16:38:31.750067 16324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "ae364743f5de4902864a43ec6f4fd96b"
format_stamp: "Formatted at 2026-03-19 16:38:31 on dist-test-slave-svn8"
server_key: "b0bb069b840f29affa21dea88ac880ec"
server_key_iv: "438b83f39404307962b0f5cd09fc969a"
server_key_version: "encryptionkey@0"
I20260319 16:38:31.751183 16324 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:38:31.751895 16339 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:31.752064 16324 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:38:31.752135 16324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "ae364743f5de4902864a43ec6f4fd96b"
format_stamp: "Formatted at 2026-03-19 16:38:31 on dist-test-slave-svn8"
server_key: "b0bb069b840f29affa21dea88ac880ec"
server_key_iv: "438b83f39404307962b0f5cd09fc969a"
server_key_version: "encryptionkey@0"
I20260319 16:38:31.752194 16324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:31.769338 16324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:31.771142 16324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:31.771272 16324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:31.775099 16324 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:46391
I20260319 16:38:31.775141 16391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:46391 every 8 connection(s)
I20260319 16:38:31.775436 16324 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:31.775982 16392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:31.777544 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16324
I20260319 16:38:31.777697 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:31.777904 25327 external_mini_cluster.cc:1468] Setting key 9a912cb1ae250385d00bf482a0e2aac6
I20260319 16:38:31.778647 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b: Bootstrap starting.
I20260319 16:38:31.779183 16392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:31.779418 16392 log.cc:826] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:31.780025 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b: No bootstrap required, opened a new log
I20260319 16:38:31.781042 16392 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae364743f5de4902864a43ec6f4fd96b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46391 } }
I20260319 16:38:31.781185 16392 raft_consensus.cc:385] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:31.781214 16392 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae364743f5de4902864a43ec6f4fd96b, State: Initialized, Role: FOLLOWER
I20260319 16:38:31.781318 16392 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae364743f5de4902864a43ec6f4fd96b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46391 } }
I20260319 16:38:31.781387 16392 raft_consensus.cc:399] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:31.781411 16392 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:31.781455 16392 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:31.782032 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:31.778402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38529 (local address 127.24.187.254:46391)
0319 16:38:31.778574 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:31.778585 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:31.778602 (+ 17us) server_negotiation.cc:408] Connection header received
0319 16:38:31.778935 (+ 333us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:31.778940 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:31.778978 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:31.779084 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:31.779489 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:31.780319 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:31.780884 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:31.781114 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:31.781479 (+ 365us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:31.781487 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:31.781487 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:31.781670 (+ 183us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:31.781689 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:31.781696 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:31.781778 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:31.781848 (+ 70us) server_negotiation.cc:300] Negotiation successful
0319 16:38:31.781885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260319 16:38:31.782050 16392 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae364743f5de4902864a43ec6f4fd96b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46391 } }
I20260319 16:38:31.782143 16392 leader_election.cc:304] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae364743f5de4902864a43ec6f4fd96b; no voters:
I20260319 16:38:31.782298 16392 leader_election.cc:290] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:31.782366 16397 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:31.782528 16392 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:31.782727 16397 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [term 1 LEADER]: Becoming Leader. State: Replica: ae364743f5de4902864a43ec6f4fd96b, State: Running, Role: LEADER
I20260319 16:38:31.782857 16397 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae364743f5de4902864a43ec6f4fd96b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46391 } }
I20260319 16:38:31.782966 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:31.783974 16398 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae364743f5de4902864a43ec6f4fd96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae364743f5de4902864a43ec6f4fd96b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46391 } } }
I20260319 16:38:31.783986 16399 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae364743f5de4902864a43ec6f4fd96b. Latest consensus state: current_term: 1 leader_uuid: "ae364743f5de4902864a43ec6f4fd96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae364743f5de4902864a43ec6f4fd96b" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 46391 } } }
I20260319 16:38:31.784065 16398 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:31.784082 16399 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b [sys.catalog]: This master's current role is: LEADER
W20260319 16:38:31.784677 16416 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:31.784725 16416 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:31.784773 16415 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:31.784956 16415 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:31.786273 16415 catalog_manager.cc:1357] Generated new cluster ID: a35af4dac8574a9d80fa4e1b76502cde
I20260319 16:38:31.786322 16415 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:31.787375 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:31.784562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50251 (local address 127.24.187.254:46391)
0319 16:38:31.784601 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:31.784603 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:31.784700 (+ 97us) server_negotiation.cc:408] Connection header received
0319 16:38:31.784798 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:31.784799 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:31.784809 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:31.784870 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:31.785170 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:31.785793 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:31.786411 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:31.786545 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:31.787129 (+ 584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:31.787133 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:31.787133 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:31.787139 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:31.787155 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:31.787159 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:31.787211 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:31.787280 (+ 69us) server_negotiation.cc:300] Negotiation successful
0319 16:38:31.787307 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:38:31.798183 16415 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:31.798736 16415 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:31.806643 16415 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ae364743f5de4902864a43ec6f4fd96b: Generated new TSK 0
I20260319 16:38:31.806838 16415 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:31.811913 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16324
2026-03-19T16:38:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:31 dist-test-slave-svn8 krb5kdc[16423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:31 dist-test-slave-svn8 krb5kdc[16423](info): set up 2 sockets
Mar 19 16:38:31 dist-test-slave-svn8 krb5kdc[16423](info): commencing operation
krb5kdc: starting...
W20260319 16:38:33.662328 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.832s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:33 dist-test-slave-svn8 krb5kdc[16423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:33.800632 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:43103
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:43103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:33.875649 16439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:33.875818 16439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:33.875838 16439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:33.877254 16439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:33.877300 16439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:33.877313 16439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:33.877324 16439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:33.877334 16439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:33.878912 16439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:43103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:33.879070 16439 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:33.879189 16439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:33.881258 16445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:33.881261 16444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:33.881356 16447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:33.881510 16439 server_base.cc:1060] running on GCE node
I20260319 16:38:33.881686 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:33.881934 16439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:33.883082 16439 hybrid_clock.cc:648] HybridClock initialized: now 1773938313883063 us; error 29 us; skew 500 ppm
Mar 19 16:38:33 dist-test-slave-svn8 krb5kdc[16423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938313, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:33.885322 16439 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:33.885831 16439 webserver.cc:492] Webserver started at http://127.24.187.254:37039/ using document root <none> and password file <none>
I20260319 16:38:33.885975 16439 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:33.886003 16439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:33.886075 16439 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:33.886912 16439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "2399ba8999a7422594fd0a3dcfdc5713"
format_stamp: "Formatted at 2026-03-19 16:38:33 on dist-test-slave-svn8"
server_key: "d22220dfb38684d64fe8a4f3afe6e4f5"
server_key_iv: "df87156587b8cac546277e6197fcf2e5"
server_key_version: "encryptionkey@0"
I20260319 16:38:33.887157 16439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "2399ba8999a7422594fd0a3dcfdc5713"
format_stamp: "Formatted at 2026-03-19 16:38:33 on dist-test-slave-svn8"
server_key: "d22220dfb38684d64fe8a4f3afe6e4f5"
server_key_iv: "df87156587b8cac546277e6197fcf2e5"
server_key_version: "encryptionkey@0"
I20260319 16:38:33.888157 16439 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:38:33.888762 16454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:33.888906 16439 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260319 16:38:33.888963 16439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "2399ba8999a7422594fd0a3dcfdc5713"
format_stamp: "Formatted at 2026-03-19 16:38:33 on dist-test-slave-svn8"
server_key: "d22220dfb38684d64fe8a4f3afe6e4f5"
server_key_iv: "df87156587b8cac546277e6197fcf2e5"
server_key_version: "encryptionkey@0"
I20260319 16:38:33.889022 16439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:33.892138 16439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:33.897924 16439 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:33.898057 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:33.901968 16439 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:43103
I20260319 16:38:33.902004 16506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:43103 every 8 connection(s)
I20260319 16:38:33.902283 16439 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:33.902796 16507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:33.904778 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16439
I20260319 16:38:33.904893 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:33.905079 25327 external_mini_cluster.cc:1468] Setting key f8080af599acaefc65c28ed985cccedf
I20260319 16:38:33.905313 16507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713: Bootstrap starting.
I20260319 16:38:33.905966 16507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:33.906219 16507 log.cc:826] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:33.906922 16507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713: No bootstrap required, opened a new log
I20260319 16:38:33.907335 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:33.905573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48651 (local address 127.24.187.254:43103)
0319 16:38:33.905764 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:33.905775 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:33.905796 (+ 21us) server_negotiation.cc:408] Connection header received
0319 16:38:33.906097 (+ 301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:33.906104 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:33.906128 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:33.906234 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:33.906910 (+ 676us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:33.906913 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:33.906913 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:33.906917 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:33.906942 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:33.906948 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:33.907060 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:33.907123 (+ 63us) server_negotiation.cc:300] Negotiation successful
0319 16:38:33.907154 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260319 16:38:33.908286 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:33.908273 16507 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2399ba8999a7422594fd0a3dcfdc5713" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43103 } }
I20260319 16:38:33.908398 16507 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260319 16:38:33.908435 16507 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2399ba8999a7422594fd0a3dcfdc5713, State: Initialized, Role: FOLLOWER
I20260319 16:38:33.908525 16507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2399ba8999a7422594fd0a3dcfdc5713" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43103 } }
I20260319 16:38:33.908591 16507 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260319 16:38:33.908614 16507 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260319 16:38:33.908658 16507 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:33.909161 16507 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2399ba8999a7422594fd0a3dcfdc5713" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43103 } }
I20260319 16:38:33.909255 16507 leader_election.cc:304] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2399ba8999a7422594fd0a3dcfdc5713; no voters:
I20260319 16:38:33.909446 16507 leader_election.cc:290] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260319 16:38:33.909495 16516 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:33.909641 16516 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [term 1 LEADER]: Becoming Leader. State: Replica: 2399ba8999a7422594fd0a3dcfdc5713, State: Running, Role: LEADER
I20260319 16:38:33.909662 16507 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:33.909771 16516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2399ba8999a7422594fd0a3dcfdc5713" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43103 } }
I20260319 16:38:33.910153 16516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2399ba8999a7422594fd0a3dcfdc5713. Latest consensus state: current_term: 1 leader_uuid: "2399ba8999a7422594fd0a3dcfdc5713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2399ba8999a7422594fd0a3dcfdc5713" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43103 } } }
I20260319 16:38:33.910215 16516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:33.910226 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2399ba8999a7422594fd0a3dcfdc5713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2399ba8999a7422594fd0a3dcfdc5713" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43103 } } }
I20260319 16:38:33.910315 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:33.911293 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:33.909861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46543 (local address 127.24.187.254:43103)
0319 16:38:33.910339 (+ 478us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:33.910341 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:33.910350 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:38:33.910369 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:33.910370 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:33.910375 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:33.910480 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:33.911037 (+ 557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:33.911038 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:33.911039 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0319 16:38:33.911042 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:33.911058 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0319 16:38:33.911062 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:33.911154 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:33.911221 (+ 67us) server_negotiation.cc:300] Negotiation successful
0319 16:38:33.911234 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
W20260319 16:38:33.911892 16533 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:33.911939 16533 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:33.911980 16524 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:33.912145 16524 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:33.913409 16524 catalog_manager.cc:1357] Generated new cluster ID: ad70c4bbc2a44b2bb9e888a501675c34
I20260319 16:38:33.913482 16524 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:33.925001 16524 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:33.925158 16524 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:33.932217 16524 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2399ba8999a7422594fd0a3dcfdc5713: Generated new TSK 0
I20260319 16:38:33.932379 16524 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:33.935720 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16439
2026-03-19T16:38:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2123 ms)
[----------] 18 tests from AuthTokenIssuingTest (38335 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:33 dist-test-slave-svn8 krb5kdc[16538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:33 dist-test-slave-svn8 krb5kdc[16538](info): set up 2 sockets
Mar 19 16:38:33 dist-test-slave-svn8 krb5kdc[16538](info): commencing operation
krb5kdc: starting...
W20260319 16:38:35.786406 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.833s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:35 dist-test-slave-svn8 krb5kdc[16538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:35.923161 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35195
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:35195,127.24.187.253:39307,127.24.187.252:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:35.996996 16554 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260319 16:38:35.997189 16554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:35.997212 16554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:35.997239 16554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:35.998797 16554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:35.998847 16554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:35.998878 16554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:35.998895 16554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:35.998908 16554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:36.000816 16554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:35195,127.24.187.253:39307,127.24.187.252:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:36.001052 16554 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:36.001209 16554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:36.003329 16562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:36.003412 16559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:36.003412 16560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:36.003713 16554 server_base.cc:1060] running on GCE node
I20260319 16:38:36.003891 16554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:36.004084 16554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:36.005214 16554 hybrid_clock.cc:648] HybridClock initialized: now 1773938316005198 us; error 24 us; skew 500 ppm
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:36.007421 16554 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:36.007864 16554 webserver.cc:492] Webserver started at http://127.24.187.254:34947/ using document root <none> and password file <none>
I20260319 16:38:36.008004 16554 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:36.008033 16554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:36.008111 16554 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:36.008992 16554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "b3e95a9456118bad1faf27f4062b3621"
server_key_iv: "b012c67ad52274f457abbbd351e2c5a9"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.009246 16554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "b3e95a9456118bad1faf27f4062b3621"
server_key_iv: "b012c67ad52274f457abbbd351e2c5a9"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.010303 16554 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:38:36.010892 16569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:36.011039 16554 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:36.011091 16554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "b3e95a9456118bad1faf27f4062b3621"
server_key_iv: "b012c67ad52274f457abbbd351e2c5a9"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.011152 16554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:36.022439 16554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:36.024019 16554 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:36.024123 16554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:36.027760 16554 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:35195
I20260319 16:38:36.027829 16621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:35195 every 8 connection(s)
I20260319 16:38:36.028069 16554 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:36.028625 16622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:36.030409 16622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:38:36.037632 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16554
I20260319 16:38:36.037777 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:36.037787 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.030997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.24.187.254:35195)
0319 16:38:36.031124 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.031130 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.031164 (+ 34us) server_negotiation.cc:408] Connection header received
0319 16:38:36.031522 (+ 358us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.031523 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.031539 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.031618 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.032204 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.032870 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.033423 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.033551 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.035397 (+ 1846us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.035405 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.035405 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.035411 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.036609 (+ 1198us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.036945 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.036948 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.036949 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.036994 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.037149 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.037150 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.037151 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.037312 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.037398 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.037530 (+ 132us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.037618 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260319 16:38:36.037838 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.030945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35195 (local address 127.0.0.1:41558)
0319 16:38:36.031079 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.031106 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.031419 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.031631 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.031633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.031900 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.032118 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.032120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.032963 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.032964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.033353 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.033355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.033459 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.033982 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.033996 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.035310 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.036675 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.036680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.036681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.036834 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.037053 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.037053 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.037053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.037089 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.037381 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.037383 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.037428 (+ 45us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.037516 (+ 88us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.037696 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":65,"thread_start_us":57,"threads_started":1}
I20260319 16:38:36.037956 25327 external_mini_cluster.cc:1468] Setting key 99c370be7c3ba18735850dde2c011c0b
I20260319 16:38:36.038792 16622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:36.039110 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.039005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33052)
0319 16:38:36.039045 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.039089 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260319 16:38:36.039223 16570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
I20260319 16:38:36.039640 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.039546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33066)
0319 16:38:36.039595 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.039619 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260319 16:38:36.039746 16622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:39307: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.187.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:36.080931 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:39307
--webserver_interface=127.24.187.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:35195,127.24.187.253:39307,127.24.187.252:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:38:36.088961 16622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } attempt: 1
I20260319 16:38:36.089401 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.089277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33082)
0319 16:38:36.089336 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.089369 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260319 16:38:36.089869 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.089698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33090)
0319 16:38:36.089823 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.089847 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260319 16:38:36.089946 16622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:39307: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
W20260319 16:38:36.155548 16631 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260319 16:38:36.155735 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:36.155757 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:36.155784 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:36.157238 16631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:36.157279 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:36.157297 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:36.157311 16631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:36.157335 16631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:36.158087 16622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } attempt: 2
I20260319 16:38:36.158496 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.158381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33092)
0319 16:38:36.158437 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.158467 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260319 16:38:36.158798 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.158720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33100)
0319 16:38:36.158755 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.158779 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260319 16:38:36.158867 16622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:39307: Network error: Client connection negotiation failed: client connection to 127.24.187.253:39307: connect: Connection refused (error 111)
I20260319 16:38:36.159093 16631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:35195,127.24.187.253:39307,127.24.187.252:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.187.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:36.159312 16631 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:36.159471 16631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:36.161679 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:36.161687 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:36.161687 16636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:36.161759 16631 server_base.cc:1060] running on GCE node
I20260319 16:38:36.162046 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:36.162232 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:36.163446 16631 hybrid_clock.cc:648] HybridClock initialized: now 1773938316163433 us; error 28 us; skew 500 ppm
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:36.165673 16631 init.cc:377] Logged in from keytab as kudu/127.24.187.253@KRBTEST.COM (short username kudu)
I20260319 16:38:36.166142 16631 webserver.cc:492] Webserver started at http://127.24.187.253:36433/ using document root <none> and password file <none>
I20260319 16:38:36.166308 16631 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:36.166348 16631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:36.166445 16631 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:36.167337 16631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data/instance:
uuid: "c70f5d80eb4542a2857bc9add89533b6"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "dbf8c33aac77f71033b3d716efb2a2ea"
server_key_iv: "bfe7381859dffc362c19fcfaef943687"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.167613 16631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/wal/instance:
uuid: "c70f5d80eb4542a2857bc9add89533b6"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "dbf8c33aac77f71033b3d716efb2a2ea"
server_key_iv: "bfe7381859dffc362c19fcfaef943687"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.168628 16631 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260319 16:38:36.169250 16646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:36.169457 16631 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:36.169518 16631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/wal
uuid: "c70f5d80eb4542a2857bc9add89533b6"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "dbf8c33aac77f71033b3d716efb2a2ea"
server_key_iv: "bfe7381859dffc362c19fcfaef943687"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.169569 16631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:36.187016 16631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:36.188763 16631 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:36.188866 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:36.192564 16631 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.253:39307
I20260319 16:38:36.192610 16698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.253:39307 every 8 connection(s)
I20260319 16:38:36.192867 16631 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
I20260319 16:38:36.193475 16699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:36.195451 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16631
I20260319 16:38:36.195338 16699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:36.195544 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-1/wal/instance
I20260319 16:38:36.195715 25327 external_mini_cluster.cc:1468] Setting key f1d2e910865ddd3a1999fd3cc59888c0
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.187.252@KRBTEST.COM; defaulting to no policy
I20260319 16:38:36.203289 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.195999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.24.187.254:35195)
0319 16:38:36.196044 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.196046 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.196204 (+ 158us) server_negotiation.cc:408] Connection header received
0319 16:38:36.196727 (+ 523us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.196728 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.196744 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.196797 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.197704 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.198178 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.198691 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.198876 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.200830 (+ 1954us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.200834 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.200834 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.200842 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.202155 (+ 1313us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.202541 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.202542 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.202542 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.202570 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.202801 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.202802 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.202802 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.202886 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.202929 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.203149 (+ 220us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.203222 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:38:36.203374 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.195971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35195 (local address 127.0.0.1:41572)
0319 16:38:36.196108 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.196135 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.196606 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.196848 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.196852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.197212 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.197606 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.197610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.198272 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.198273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.198620 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.198622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.198744 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.199316 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.199332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.200731 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.202234 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.202240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.202242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.202473 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.202648 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.202649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.202650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.202732 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.202938 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.202942 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.203045 (+ 103us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.203132 (+ 87us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.203231 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260319 16:38:36.203887 16699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.252@KRBTEST.COM" created.
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:38:36.210155 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.204110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33114)
0319 16:38:36.204152 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.204162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.204202 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.204429 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.204430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.204562 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.204696 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.204698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.205468 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.205469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.205886 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.205889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.205954 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.206325 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.206336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.207678 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.209341 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.209342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.209342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.209458 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.209618 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.209619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.209619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.209646 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.209863 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.209863 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.209900 (+ 37us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.209959 (+ 59us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.210055 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260319 16:38:36.210309 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.204171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.24.187.253:39307)
0319 16:38:36.204317 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.204322 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.204337 (+ 15us) server_negotiation.cc:408] Connection header received
0319 16:38:36.204354 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.204356 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.204365 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.204421 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.204761 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.205404 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.205957 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.206140 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.207761 (+ 1621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.207768 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.207768 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.207774 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.209230 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.209538 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.209540 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.209541 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.209567 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.209690 (+ 123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.209691 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.209691 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.209811 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.209884 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.209985 (+ 101us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.210117 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260319 16:38:36.210835 16699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:36.211202 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.211085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46228)
0319 16:38:36.211131 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.211176 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260319 16:38:36.211362 16648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
I20260319 16:38:36.211771 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.211671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46236)
0319 16:38:36.211721 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.211747 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260319 16:38:36.211902 16699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33059: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:36.244808 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:33059
--webserver_interface=127.24.187.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:35195,127.24.187.253:39307,127.24.187.252:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:38:36.249996 16622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } attempt: 3
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:38:36.256299 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.250390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33118)
0319 16:38:36.250444 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.250457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.250520 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.250684 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.250685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.250830 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.250987 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.250989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.251897 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.251898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.252239 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.252240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.252290 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.252709 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.252714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.253881 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.255500 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.255501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.255501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.255632 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.255796 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.255798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.255798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.255831 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.256047 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.256047 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.256088 (+ 41us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.256137 (+ 49us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.256218 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:38:36.256343 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.250465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.24.187.253:39307)
0319 16:38:36.250534 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.250537 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.250566 (+ 29us) server_negotiation.cc:408] Connection header received
0319 16:38:36.250597 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.250598 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.250611 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.250663 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.251071 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.251800 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.252309 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.252495 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.253966 (+ 1471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.253970 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.253971 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.253978 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.255421 (+ 1443us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.255692 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.255693 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.255693 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.255718 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.255891 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.255892 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.255893 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.255997 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.256058 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.256164 (+ 106us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.256240 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260319 16:38:36.256767 16622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:36.257074 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.256982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46238)
0319 16:38:36.257025 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.257056 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:38:36.257364 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.257288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46248)
0319 16:38:36.257327 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.257347 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260319 16:38:36.257444 16622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33059: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
I20260319 16:38:36.261086 16699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } attempt: 1
I20260319 16:38:36.261423 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.261323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46252)
0319 16:38:36.261366 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.261397 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:38:36.261790 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.261693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46264)
0319 16:38:36.261748 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.261771 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260319 16:38:36.261920 16699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33059: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
I20260319 16:38:36.288570 16622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } attempt: 1
I20260319 16:38:36.288980 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.288869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46278)
0319 16:38:36.288921 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.288951 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260319 16:38:36.289336 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.289248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46280)
0319 16:38:36.289291 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.289315 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260319 16:38:36.289417 16622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33059: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
W20260319 16:38:36.320829 16708 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260319 16:38:36.321007 16708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:36.321022 16708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:36.321050 16708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:36.322592 16708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:36.322638 16708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:36.322650 16708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:36.322661 16708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:36.322672 16708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:36.324298 16708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:35195,127.24.187.253:39307,127.24.187.252:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:33059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.187.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:36.324486 16708 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:36.324657 16708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:36.326727 16716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:36.326757 16714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:36.326821 16713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:36.327008 16708 server_base.cc:1060] running on GCE node
I20260319 16:38:36.327150 16708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:36.327337 16708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:36.328482 16708 hybrid_clock.cc:648] HybridClock initialized: now 1773938316328468 us; error 24 us; skew 500 ppm
I20260319 16:38:36.330080 16699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } attempt: 2
I20260319 16:38:36.330423 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.330317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46286)
0319 16:38:36.330369 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.330395 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:36.330763 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.330664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46292)
0319 16:38:36.330710 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.330738 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260319 16:38:36.330849 16699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:33059: Network error: Client connection negotiation failed: client connection to 127.24.187.252:33059: connect: Connection refused (error 111)
I20260319 16:38:36.330883 16708 init.cc:377] Logged in from keytab as kudu/127.24.187.252@KRBTEST.COM (short username kudu)
I20260319 16:38:36.331326 16708 webserver.cc:492] Webserver started at http://127.24.187.252:35137/ using document root <none> and password file <none>
I20260319 16:38:36.331466 16708 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:36.331502 16708 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:36.331578 16708 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:36.332427 16708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data/instance:
uuid: "afaae714b0e94498928bf0c4ca4e3518"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "c4cff9a9f67bb9d328d917d57283468a"
server_key_iv: "50cd1f4c36c4a596c85df3dd67a8118c"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.332686 16708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/wal/instance:
uuid: "afaae714b0e94498928bf0c4ca4e3518"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "c4cff9a9f67bb9d328d917d57283468a"
server_key_iv: "50cd1f4c36c4a596c85df3dd67a8118c"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.333690 16708 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260319 16:38:36.334299 16723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:36.334444 16708 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260319 16:38:36.334518 16708 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/wal
uuid: "afaae714b0e94498928bf0c4ca4e3518"
format_stamp: "Formatted at 2026-03-19 16:38:36 on dist-test-slave-svn8"
server_key: "c4cff9a9f67bb9d328d917d57283468a"
server_key_iv: "50cd1f4c36c4a596c85df3dd67a8118c"
server_key_version: "encryptionkey@0"
I20260319 16:38:36.334573 16708 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:36.355935 16708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:36.357820 16708 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:36.357950 16708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:36.361660 16708 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.252:33059
I20260319 16:38:36.361739 16775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.252:33059 every 8 connection(s)
I20260319 16:38:36.362011 16708 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
I20260319 16:38:36.362535 16776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:36.364545 16622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } attempt: 2
I20260319 16:38:36.364593 16776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:38:36.369365 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16708
I20260319 16:38:36.369470 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1773937967898507-25327-0/minicluster-data/master-2/wal/instance
I20260319 16:38:36.369625 25327 external_mini_cluster.cc:1468] Setting key eee5d383dc5193f902f33dff58a96ca0
I20260319 16:38:36.372282 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.364878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46306)
0319 16:38:36.364944 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.364955 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.364998 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.365727 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.365728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.365888 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.366016 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.366018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.366968 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.366969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.367282 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.367282 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.367340 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.367745 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.367750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.369007 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.371084 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.371086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.371396 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.371587 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.371588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.371618 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.371968 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.371968 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.372025 (+ 57us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.372123 (+ 98us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.372214 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260319 16:38:36.372496 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.365257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.24.187.254:35195)
0319 16:38:36.365313 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.365315 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.365439 (+ 124us) server_negotiation.cc:408] Connection header received
0319 16:38:36.365691 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.365692 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.365703 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.365789 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.366487 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.366897 (+ 410us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.367382 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.367961 (+ 579us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.369812 (+ 1851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.369816 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.369816 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.369823 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.371489 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371827 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.371827 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.371828 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.371844 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371988 (+ 144us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.371989 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.371989 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.372100 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.372161 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.372354 (+ 193us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.372432 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260319 16:38:36.372658 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.365205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:35195 (local address 127.0.0.1:41578)
0319 16:38:36.365338 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.365366 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.365612 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.365811 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.365813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.366090 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.366410 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.366413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.366963 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.366964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.367313 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.367314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.367426 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.367935 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.367951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.369671 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.371553 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.371560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.371761 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.371892 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.371893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.371919 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.372159 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.372164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.372234 (+ 70us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.372330 (+ 96us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.372450 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260319 16:38:36.372931 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.365019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.24.187.252:33059)
0319 16:38:36.365176 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.365185 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.365206 (+ 21us) server_negotiation.cc:408] Connection header received
0319 16:38:36.365599 (+ 393us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.365607 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.365631 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.365702 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.366087 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.366900 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.367345 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.367537 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.369083 (+ 1546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.369090 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.369091 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.369097 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.371001 (+ 1904us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371465 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.371468 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.371469 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.371507 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.371670 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.371671 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.371671 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.371894 (+ 223us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.372722 (+ 828us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.372767 (+ 45us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.372843 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260319 16:38:36.373536 16776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:38:36.374462 16622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1: Bootstrap starting.
I20260319 16:38:36.375223 16622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:36.375494 16622 log.cc:826] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:36.376241 16622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1: No bootstrap required, opened a new log
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:38:36.377787 16622 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.377877 16622 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b442abb8aa4cc9b770f1c16ad34cf1, State: Initialized, Role: FOLLOWER
I20260319 16:38:36.377919 16622 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.378140 16622 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:36.378252 16783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.378329 16783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:36.378464 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.369956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.24.187.252:33059)
0319 16:38:36.370298 (+ 342us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.370299 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.370308 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:38:36.370322 (+ 14us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.370323 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.370331 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.370382 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.371179 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.371893 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.372680 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.372852 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.374963 (+ 2111us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.374966 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.374967 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.374973 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.376918 (+ 1945us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.377246 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.377247 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.377248 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.377274 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.377428 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.377428 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.377429 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.377528 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.377655 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.378078 (+ 423us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.378162 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":78,"threads_started":1}
I20260319 16:38:36.379622 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.373803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:39307 (local address 127.0.0.1:33124)
0319 16:38:36.373927 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.373937 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.373987 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.374112 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.374113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.374249 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.374434 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.374436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.375220 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.375221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.375617 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.375619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.375677 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.376013 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.376017 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.377209 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.378789 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.378790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.378790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.378913 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.379062 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.379063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.379063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.379095 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.379358 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.379359 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.379400 (+ 41us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.379457 (+ 57us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.379539 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260319 16:38:36.379652 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.373857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.24.187.253:39307)
0319 16:38:36.373921 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.373923 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.373973 (+ 50us) server_negotiation.cc:408] Connection header received
0319 16:38:36.374035 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.374037 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.374047 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.374102 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.374526 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.375164 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.375687 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.375850 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.377281 (+ 1431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.377285 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.377285 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.377290 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.378705 (+ 1415us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.378971 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.378972 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.378972 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.378994 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.379158 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.379159 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.379159 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.379264 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.379331 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.379493 (+ 162us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.379563 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260319 16:38:36.379925 16776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } has no permanent_uuid. Determining permanent_uuid...
W20260319 16:38:36.380419 16794 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:36.380652 16794 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:36.380944 16554 master_runner.cc:186] Error getting master registration for 127.24.187.253:39307: OK, instance_id { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" instance_seqno: 1773938316191529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260319 16:38:36.382561 16554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:38:36.385646 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.380133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46334)
0319 16:38:36.380169 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.380178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.380210 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.380503 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.380504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.380639 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.380797 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.380798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.381565 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.381566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.382073 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.382075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.382134 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.382507 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.382511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.383634 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.384925 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.384926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.384926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.385058 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.385179 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.385180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.385180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.385204 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.385384 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.385385 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.385419 (+ 34us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.385470 (+ 51us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.385561 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260319 16:38:36.386003 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.380172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.24.187.252:33059)
0319 16:38:36.380290 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.380292 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.380299 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:38:36.380316 (+ 17us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.380317 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.380337 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.380386 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.380873 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.381485 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.382270 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.382439 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.383703 (+ 1264us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.383705 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.383706 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.383711 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.384855 (+ 1144us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.385123 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.385124 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.385124 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.385140 (+ 16us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.385249 (+ 109us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.385250 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.385250 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.385337 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.385387 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.385492 (+ 105us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.385637 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":98}
I20260319 16:38:36.387172 16776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: Bootstrap starting.
I20260319 16:38:36.387696 16776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:36.387918 16776 log.cc:826] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:36.388249 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.380430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.24.187.254:35195)
0319 16:38:36.380465 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.380467 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.380474 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:38:36.380492 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.380493 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.380501 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.380543 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.381242 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.381920 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.383045 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.383223 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.385493 (+ 2270us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.385496 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.385497 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.385504 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.386948 (+ 1444us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.387275 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.387276 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.387277 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.387299 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.387447 (+ 148us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.387448 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.387448 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.387547 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.387670 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.388081 (+ 411us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.388161 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:36.388562 16776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: No bootstrap required, opened a new log
I20260319 16:38:36.389616 16776 raft_consensus.cc:359] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.389755 16776 raft_consensus.cc:740] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afaae714b0e94498928bf0c4ca4e3518, State: Initialized, Role: FOLLOWER
I20260319 16:38:36.389861 16776 consensus_queue.cc:260] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.390163 16776 sys_catalog.cc:565] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:36.390134 16797 sys_catalog.cc:455] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.391451 16797 sys_catalog.cc:458] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [sys.catalog]: This master's current role is: FOLLOWER
W20260319 16:38:36.391781 16808 catalog_manager.cc:1568] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:36.391867 16808 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:38:36.393199 16708 master_runner.cc:186] Error getting master registration for 127.24.187.253:39307: OK, instance_id { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" instance_seqno: 1773938316191529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260319 16:38:36.395185 16708 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260319 16:38:36.398164 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.390195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.24.187.253:39307)
0319 16:38:36.390232 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.390234 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.390349 (+ 115us) server_negotiation.cc:408] Connection header received
0319 16:38:36.390372 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.390373 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.390383 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.390425 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.391172 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.391811 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.392805 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.392978 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.395218 (+ 2240us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.395221 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.395222 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.395228 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.396835 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.397162 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.397163 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.397164 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.397187 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.397330 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.397331 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.397332 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.397436 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.397552 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.397976 (+ 424us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.398070 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:36.399344 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260319 16:38:36.400241 16596 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41592
I20260319 16:38:36.400369 16596 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260319 16:38:36.400425 16596 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:36.401649 16596 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.401881 16596 leader_election.cc:290] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [CANDIDATE]: Term 1 election: Requested vote from peers c70f5d80eb4542a2857bc9add89533b6 (127.24.187.253:39307), afaae714b0e94498928bf0c4ca4e3518 (127.24.187.252:33059)
I20260319 16:38:36.402456 16750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "afaae714b0e94498928bf0c4ca4e3518"
I20260319 16:38:36.402567 16750 raft_consensus.cc:3060] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:36.403672 16750 raft_consensus.cc:2468] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b442abb8aa4cc9b770f1c16ad34cf1 in term 1.
I20260319 16:38:36.403973 16571 leader_election.cc:304] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afaae714b0e94498928bf0c4ca4e3518, d7b442abb8aa4cc9b770f1c16ad34cf1; no voters:
I20260319 16:38:36.404109 16783 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:36.404273 16783 raft_consensus.cc:697] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [term 1 LEADER]: Becoming Leader. State: Replica: d7b442abb8aa4cc9b770f1c16ad34cf1, State: Running, Role: LEADER
I20260319 16:38:36.404359 16783 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.404806 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b442abb8aa4cc9b770f1c16ad34cf1. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.404934 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:36.405241 16816 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:36.405473 16816 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:36.407051 16750 raft_consensus.cc:1275] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [term 1 FOLLOWER]: Refusing update from remote peer d7b442abb8aa4cc9b770f1c16ad34cf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:38:36.407352 16814 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:38:36.408398 16797 sys_catalog.cc:455] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b442abb8aa4cc9b770f1c16ad34cf1. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.408473 16797 sys_catalog.cc:458] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:36.409816 16783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.409900 16783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:36.410125 16818 mvcc.cc:204] Tried to move back new op lower bound from 7266051344000618496 to 7266051343992922112. Current Snapshot: MvccSnapshot[applied={T|T < 7266051344000618496}]
I20260319 16:38:36.410228 16797 sys_catalog.cc:455] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.410293 16797 sys_catalog.cc:458] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:36.414345 16816 catalog_manager.cc:1357] Generated new cluster ID: 70e751157e484b2f8d304cfbfaa99645
I20260319 16:38:36.414391 16816 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:36.422088 16699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } attempt: 3
Mar 19 16:38:36 dist-test-slave-svn8 krb5kdc[16538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938316, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:38:36.428517 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.413986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.24.187.253:39307)
0319 16:38:36.414028 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.414030 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.414636 (+ 606us) server_negotiation.cc:408] Connection header received
0319 16:38:36.414767 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.414768 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.414777 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.414805 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.416240 (+ 1435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.416926 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.418026 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.418207 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.420436 (+ 2229us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.420441 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.420441 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.420449 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.422007 (+ 1558us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.427593 (+ 5586us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.427594 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.427595 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.427633 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.427849 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.427850 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.427850 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.427978 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.428045 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.428329 (+ 284us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.428417 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:36.430202 16816 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:36.430974 16816 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:36.432448 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.422317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:33059 (local address 127.0.0.1:46356)
0319 16:38:36.422406 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:36.422426 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:36.422461 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:36.422868 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:36.422868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:36.423033 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:36.423214 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.423216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.424158 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.424159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:36.424707 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:36.424708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.424770 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.425220 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:36.425240 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:36.427060 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:36.429609 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.429610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.429611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.429804 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.431869 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:36.431870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:36.431871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:36.431915 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:36.432161 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:36.432162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:36.432212 (+ 50us) client_negotiation.cc:769] Sending connection context
0319 16:38:36.432262 (+ 50us) client_negotiation.cc:240] Negotiation successful
0319 16:38:36.432355 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260319 16:38:36.432780 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.422544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.24.187.252:33059)
0319 16:38:36.422598 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.422600 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.422608 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:38:36.422628 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.422628 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.422637 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.422682 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.423311 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.424052 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.424934 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.425134 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.427181 (+ 2047us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.427184 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.427184 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.427192 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.429529 (+ 2337us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.429902 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.429904 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.429904 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.429926 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.431976 (+ 2050us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.431977 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.431977 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.432085 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.432132 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.432605 (+ 473us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.432698 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:38:36.434161 16699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: Bootstrap starting.
I20260319 16:38:36.435078 16699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:36.435329 16699 log.cc:826] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:36.436878 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.414398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.24.187.254:35195)
0319 16:38:36.414435 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.414437 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.414478 (+ 41us) server_negotiation.cc:408] Connection header received
0319 16:38:36.415666 (+ 1188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.415667 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.415680 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.415708 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.416511 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.417139 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.419157 (+ 2018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.419345 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.427575 (+ 8230us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.427580 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.427580 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.427588 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.429197 (+ 1609us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.430018 (+ 821us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.430019 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.430019 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.430057 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.436096 (+ 6039us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.436097 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.436097 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.436203 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.436297 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.436696 (+ 399us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.436789 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260319 16:38:36.438839 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:36.413978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.24.187.252:33059)
0319 16:38:36.414028 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.414030 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.414384 (+ 354us) server_negotiation.cc:408] Connection header received
0319 16:38:36.414581 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.414583 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.414592 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.414636 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.417862 (+ 3226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.418484 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:36.425786 (+ 7302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:36.425946 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:36.430118 (+ 4172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:36.430121 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:36.430121 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:36.430127 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:36.431774 (+ 1647us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.437384 (+ 5610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.437385 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.437386 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.437425 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:36.438203 (+ 778us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:36.438217 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:36.438217 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:36.438329 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:36.438407 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:36.438665 (+ 258us) server_negotiation.cc:300] Negotiation successful
0319 16:38:36.438755 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:38:36.440636 16699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: No bootstrap required, opened a new log
I20260319 16:38:36.442204 16699 raft_consensus.cc:359] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.442327 16699 raft_consensus.cc:740] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70f5d80eb4542a2857bc9add89533b6, State: Initialized, Role: FOLLOWER
I20260319 16:38:36.442380 16699 consensus_queue.cc:260] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } }
I20260319 16:38:36.442657 16699 sys_catalog.cc:565] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:36.442728 16673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c70f5d80eb4542a2857bc9add89533b6"
I20260319 16:38:36.442806 16673 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:36.443811 16673 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b442abb8aa4cc9b770f1c16ad34cf1 in term 1.
I20260319 16:38:36.444072 16672 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [term 1 FOLLOWER]: Refusing update from remote peer d7b442abb8aa4cc9b770f1c16ad34cf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:38:36.445367 16814 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:38:36.447047 16816 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1: Generated new TSK 0
I20260319 16:38:36.447424 16826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.447577 16826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:36.447667 16826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b442abb8aa4cc9b770f1c16ad34cf1. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.447767 16826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:36.447837 16826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.447925 16826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:36.448143 16816 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:36.448809 16836 mvcc.cc:204] Tried to move back new op lower bound from 7266051344083009536 to 7266051343992922112. Current Snapshot: MvccSnapshot[applied={T|T < 7266051344000618496}]
I20260319 16:38:36.449574 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b442abb8aa4cc9b770f1c16ad34cf1" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 35195 } } peers { permanent_uuid: "c70f5d80eb4542a2857bc9add89533b6" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 39307 } } peers { permanent_uuid: "afaae714b0e94498928bf0c4ca4e3518" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 33059 } } }
I20260319 16:38:36.449658 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b442abb8aa4cc9b770f1c16ad34cf1 [sys.catalog]: This master's current role is: LEADER
W20260319 16:38:36.450011 16842 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:36.451444 16842 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260319 16:38:36.467231 16625 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.465550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.24.187.254:35195)
0319 16:38:36.465597 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.465599 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.466560 (+ 961us) server_negotiation.cc:408] Connection header received
0319 16:38:36.466670 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.466671 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.466683 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.466712 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.467162 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41604: BlockingRecv error: recv got EOF from 127.0.0.1:41604 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260319 16:38:36.468169 16780 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.465951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.24.187.252:33059)
0319 16:38:36.465990 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.465992 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.467646 (+ 1654us) server_negotiation.cc:408] Connection header received
0319 16:38:36.467744 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.467745 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.467757 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.467787 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.468113 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46358: BlockingRecv error: recv got EOF from 127.0.0.1:46358 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260319 16:38:36.468972 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.466274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.24.187.253:39307)
0319 16:38:36.466313 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.466315 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.468516 (+ 2201us) server_negotiation.cc:408] Connection header received
0319 16:38:36.468609 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.468611 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.468621 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.468649 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.468920 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33156: BlockingRecv error: recv got EOF from 127.0.0.1:33156 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260319 16:38:36.471359 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.470620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.24.187.253:39307)
0319 16:38:36.470659 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.470661 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.470861 (+ 200us) server_negotiation.cc:408] Connection header received
0319 16:38:36.470958 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.470959 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.470973 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.471000 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.471302 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33158: BlockingRecv error: recv got EOF from 127.0.0.1:33158 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260319 16:38:36.474428 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.473783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.24.187.253:39307)
0319 16:38:36.473815 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.473816 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.473985 (+ 169us) server_negotiation.cc:408] Connection header received
0319 16:38:36.474071 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.474072 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.474081 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.474107 (+ 26us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.474386 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33172: BlockingRecv error: recv got EOF from 127.0.0.1:33172 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260319 16:38:36.478304 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.477601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.24.187.253:39307)
0319 16:38:36.477632 (+ 31us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.477633 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.477851 (+ 218us) server_negotiation.cc:408] Connection header received
0319 16:38:36.477939 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.477940 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.477949 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.477975 (+ 26us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.478264 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33184: BlockingRecv error: recv got EOF from 127.0.0.1:33184 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260319 16:38:36.484208 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.483514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.24.187.253:39307)
0319 16:38:36.483547 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.483548 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.483749 (+ 201us) server_negotiation.cc:408] Connection header received
0319 16:38:36.483837 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.483838 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.483847 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.483873 (+ 26us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.484164 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33190: BlockingRecv error: recv got EOF from 127.0.0.1:33190 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260319 16:38:36.494403 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.493547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.24.187.253:39307)
0319 16:38:36.493584 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.493586 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.493879 (+ 293us) server_negotiation.cc:408] Connection header received
0319 16:38:36.493976 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.493978 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.493988 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.494016 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.494338 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33198: BlockingRecv error: recv got EOF from 127.0.0.1:33198 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260319 16:38:36.512851 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.511972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.24.187.253:39307)
0319 16:38:36.512013 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.512015 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.512251 (+ 236us) server_negotiation.cc:408] Connection header received
0319 16:38:36.512367 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.512368 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.512380 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.512409 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.512787 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33206: BlockingRecv error: recv got EOF from 127.0.0.1:33206 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260319 16:38:36.547467 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.546522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.24.187.253:39307)
0319 16:38:36.546567 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.546570 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.546847 (+ 277us) server_negotiation.cc:408] Connection header received
0319 16:38:36.546950 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.546952 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.546964 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.546993 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.547404 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33214: BlockingRecv error: recv got EOF from 127.0.0.1:33214 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260319 16:38:36.613633 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.612955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.24.187.253:39307)
0319 16:38:36.612998 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.613000 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.613114 (+ 114us) server_negotiation.cc:408] Connection header received
0319 16:38:36.613180 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.613182 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.613194 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.613246 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.613569 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33220: BlockingRecv error: recv got EOF from 127.0.0.1:33220 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260319 16:38:36.743465 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:36.742792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.24.187.253:39307)
0319 16:38:36.742847 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:36.742850 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:36.742946 (+ 96us) server_negotiation.cc:408] Connection header received
0319 16:38:36.743023 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:36.743024 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:36.743036 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:36.743080 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:36.743406 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33482: BlockingRecv error: recv got EOF from 127.0.0.1:33482 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260319 16:38:37.001312 16703 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:37.000773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.24.187.253:39307)
0319 16:38:37.000829 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:37.000831 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:37.000840 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:38:37.000884 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:37.000886 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:37.000896 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:37.000938 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:37.001241 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33484: BlockingRecv error: recv got EOF from 127.0.0.1:33484 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260319 16:38:37.392375 16808 catalog_manager.cc:1269] Loaded cluster ID: 70e751157e484b2f8d304cfbfaa99645
I20260319 16:38:37.392460 16808 catalog_manager.cc:1562] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: loading cluster ID for follower catalog manager: success
I20260319 16:38:37.393370 16808 catalog_manager.cc:1584] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: acquiring CA information for follower catalog manager: success
I20260319 16:38:37.393657 16808 catalog_manager.cc:1612] T 00000000000000000000000000000000 P afaae714b0e94498928bf0c4ca4e3518: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260319 16:38:37.451871 16842 catalog_manager.cc:1269] Loaded cluster ID: 70e751157e484b2f8d304cfbfaa99645
I20260319 16:38:37.451946 16842 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: loading cluster ID for follower catalog manager: success
I20260319 16:38:37.452888 16842 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: acquiring CA information for follower catalog manager: success
I20260319 16:38:37.453177 16842 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c70f5d80eb4542a2857bc9add89533b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260319 16:38:37.516850 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:37.514635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.24.187.253:39307)
0319 16:38:37.514763 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:37.514765 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:37.514776 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:38:37.514806 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:37.514807 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:37.514847 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:37.514911 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:37.515199 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.515710 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:37.516294 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.516430 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:37.516477 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:37.516587 (+ 110us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:37.516645 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:37.516746 (+ 101us) server_negotiation.cc:300] Negotiation successful
0319 16:38:37.516766 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260319 16:38:37.520749 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:37.518739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.24.187.252:33059)
0319 16:38:37.518866 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:37.518869 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:37.518878 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:38:37.518953 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:37.518953 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:37.518977 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:37.519031 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:37.519246 (+ 215us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.519754 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:37.520268 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.520411 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:37.520446 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:37.520529 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:37.520595 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:37.520652 (+ 57us) server_negotiation.cc:300] Negotiation successful
0319 16:38:37.520675 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":51,"threads_started":1}
I20260319 16:38:37.524720 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:37.522299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.24.187.253:39307)
0319 16:38:37.522358 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:37.522360 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:37.522546 (+ 186us) server_negotiation.cc:408] Connection header received
0319 16:38:37.522639 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:37.522641 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:37.522656 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:37.522725 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:37.523026 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.523675 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:37.524286 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.524444 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:37.524465 (+ 21us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:37.524496 (+ 31us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:37.524549 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:37.524622 (+ 73us) server_negotiation.cc:300] Negotiation successful
0319 16:38:37.524645 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260319 16:38:37.524796 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:37.522327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.24.187.252:33059)
0319 16:38:37.522359 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:37.522360 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:37.522368 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:38:37.522427 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:37.522428 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:37.522439 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:37.522489 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:37.522785 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.523474 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:37.524328 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.524513 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:37.524546 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:37.524573 (+ 27us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:37.524618 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:37.524709 (+ 91us) server_negotiation.cc:300] Negotiation successful
0319 16:38:37.524729 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:37.525360 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:37.522266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.24.187.254:35195)
0319 16:38:37.522533 (+ 267us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:37.522535 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:37.522770 (+ 235us) server_negotiation.cc:408] Connection header received
0319 16:38:37.522874 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:37.522876 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:37.522910 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:37.522990 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:37.523253 (+ 263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.523896 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:37.524634 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:37.524807 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:37.524864 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:37.524960 (+ 96us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:37.525027 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:37.525252 (+ 225us) server_negotiation.cc:300] Negotiation successful
0319 16:38:37.525285 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":53,"threads_started":1}
I20260319 16:38:37.526924 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16554
I20260319 16:38:37.531982 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16631
I20260319 16:38:37.536353 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16708
2026-03-19T16:38:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3601 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Mar 19 16:38:37 dist-test-slave-svn8 krb5kdc[16928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Mar 19 16:38:37 dist-test-slave-svn8 krb5kdc[16928](info): set up 2 sockets
Mar 19 16:38:37 dist-test-slave-svn8 krb5kdc[16928](info): commencing operation
krb5kdc: starting...
W20260319 16:38:39.378590 25327 mini_kdc.cc:121] Time spent starting KDC: real 1.824s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/test-user.keytab.
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-03-19T16:38:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-19T16:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:39.517539 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:43017
--webserver_interface=127.24.187.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:43017,127.24.187.253:42427,127.24.187.252:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260319 16:38:39.591924 16944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260319 16:38:39.592109 16944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:39.592131 16944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:39.592156 16944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:39.593652 16944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:39.593694 16944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:39.593734 16944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:39.593750 16944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:39.593765 16944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:39.595455 16944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:43017,127.24.187.253:42427,127.24.187.252:45249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.254:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.187.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:39.595665 16944 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:39.595827 16944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:39.598042 16950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:39.598078 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:39.598248 16944 server_base.cc:1060] running on GCE node
W20260319 16:38:39.598050 16949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:39.598536 16944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:39.598745 16944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:39.599900 16944 hybrid_clock.cc:648] HybridClock initialized: now 1773938319599885 us; error 30 us; skew 500 ppm
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:39.602356 16944 init.cc:377] Logged in from keytab as kudu/127.24.187.254@KRBTEST.COM (short username kudu)
I20260319 16:38:39.602859 16944 webserver.cc:492] Webserver started at http://127.24.187.254:43737/ using document root <none> and password file <none>
I20260319 16:38:39.603019 16944 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:39.603055 16944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:39.603278 16944 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:39.604188 16944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data/instance:
uuid: "58bf5b5ee5274473812dd2d4b926d0d4"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "d0426a6a8a1ece515122946a349339b4"
server_key_iv: "30706cacc1981fd37c0c743cd20290c4"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.604480 16944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/wal/instance:
uuid: "58bf5b5ee5274473812dd2d4b926d0d4"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "d0426a6a8a1ece515122946a349339b4"
server_key_iv: "30706cacc1981fd37c0c743cd20290c4"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.605511 16944 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260319 16:38:39.606129 16959 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:39.606336 16944 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:39.606398 16944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/wal
uuid: "58bf5b5ee5274473812dd2d4b926d0d4"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "d0426a6a8a1ece515122946a349339b4"
server_key_iv: "30706cacc1981fd37c0c743cd20290c4"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.606458 16944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:39.615849 16944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:39.622606 16944 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:39.622733 16944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:39.626423 16944 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.254:43017
I20260319 16:38:39.626469 17011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.254:43017 every 8 connection(s)
I20260319 16:38:39.626725 16944 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/data/info.pb
I20260319 16:38:39.627317 17012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:39.629052 17012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.631974 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 16944
I20260319 16:38:39.632055 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-0/wal/instance
I20260319 16:38:39.632195 25327 external_mini_cluster.cc:1468] Setting key fa684040a034e47b7b08be401eb9139e
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:38:39.636896 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.629731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.24.187.254:43017)
0319 16:38:39.629881 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.629886 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.629905 (+ 19us) server_negotiation.cc:408] Connection header received
0319 16:38:39.630263 (+ 358us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.630265 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.630281 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.630354 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.630953 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.631631 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.632194 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.632423 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.634242 (+ 1819us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.634250 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.634250 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.634256 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.635670 (+ 1414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.636064 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.636067 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.636067 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.636105 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.636294 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.636295 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.636295 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.636442 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.636501 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.636686 (+ 185us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.636781 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260319 16:38:39.637022 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.629651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43017 (local address 127.0.0.1:37852)
0319 16:38:39.629801 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.629834 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.630164 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.630368 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.630371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.630637 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.630859 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.630862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.631718 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.631719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.632112 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.632114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.632232 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.632728 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.632741 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.634159 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.635746 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.635753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.635755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.635980 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.636163 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.636164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.636165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.636230 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.636500 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.636504 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.636574 (+ 70us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.636672 (+ 98us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.636881 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":79,"thread_start_us":80,"threads_started":1}
I20260319 16:38:39.637619 17012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.637923 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.637809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43568)
0319 16:38:39.637851 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.637898 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260319 16:38:39.638027 16960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
I20260319 16:38:39.638540 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.638448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43578)
0319 16:38:39.638492 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.638518 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260319 16:38:39.638638 17012 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:42427: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.187.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:39.677495 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:42427
--webserver_interface=127.24.187.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:43017,127.24.187.253:42427,127.24.187.252:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:38:39.687836 17012 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } attempt: 1
I20260319 16:38:39.688308 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.688160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43586)
0319 16:38:39.688220 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.688266 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260319 16:38:39.688741 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.688607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43588)
0319 16:38:39.688674 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.688709 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260319 16:38:39.688851 17012 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:42427: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
W20260319 16:38:39.753342 17021 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260319 16:38:39.753520 17021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:39.753535 17021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:39.753556 17021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:39.755084 17021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:39.755136 17021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:39.755149 17021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:39.755160 17021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:39.755170 17021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:39.756784 17021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:43017,127.24.187.253:42427,127.24.187.252:45249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.253:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.187.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:39.756965 17021 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:39.757030 17012 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } attempt: 2
I20260319 16:38:39.757120 17021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260319 16:38:39.757448 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.757303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43594)
0319 16:38:39.757363 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.757410 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260319 16:38:39.757869 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.757781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43596)
0319 16:38:39.757813 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.757842 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260319 16:38:39.757995 17012 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.253:42427: Network error: Client connection negotiation failed: client connection to 127.24.187.253:42427: connect: Connection refused (error 111)
W20260319 16:38:39.759413 17026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:39.759445 17029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:39.759536 17021 server_base.cc:1060] running on GCE node
W20260319 16:38:39.759441 17027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:39.759757 17021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:39.759965 17021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:39.761119 17021 hybrid_clock.cc:648] HybridClock initialized: now 1773938319761103 us; error 29 us; skew 500 ppm
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:39.763592 17021 init.cc:377] Logged in from keytab as kudu/127.24.187.253@KRBTEST.COM (short username kudu)
I20260319 16:38:39.764102 17021 webserver.cc:492] Webserver started at http://127.24.187.253:33719/ using document root <none> and password file <none>
I20260319 16:38:39.764273 17021 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:39.764309 17021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:39.764408 17021 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:39.765308 17021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data/instance:
uuid: "4400ceb324334dbe8794262dbb34fec1"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "5b1efd584a88603bc81cb2672a2d54ea"
server_key_iv: "0124d2cf86aed551a373292f3e1e400f"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.765611 17021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/wal/instance:
uuid: "4400ceb324334dbe8794262dbb34fec1"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "5b1efd584a88603bc81cb2672a2d54ea"
server_key_iv: "0124d2cf86aed551a373292f3e1e400f"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.766791 17021 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:38:39.767592 17036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:39.767834 17021 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260319 16:38:39.767910 17021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/wal
uuid: "4400ceb324334dbe8794262dbb34fec1"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "5b1efd584a88603bc81cb2672a2d54ea"
server_key_iv: "0124d2cf86aed551a373292f3e1e400f"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.767971 17021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:39.790832 17021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:39.792732 17021 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:39.792861 17021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:39.796694 17021 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.253:42427
I20260319 16:38:39.796741 17088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.253:42427 every 8 connection(s)
I20260319 16:38:39.797022 17021 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/data/info.pb
I20260319 16:38:39.797587 17089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:39.799394 17089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.802129 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 17021
I20260319 16:38:39.802227 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-1/wal/instance
I20260319 16:38:39.802384 25327 external_mini_cluster.cc:1468] Setting key 7134d77260a24a11e236984d00077ec0
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
I20260319 16:38:39.806984 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.800034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.24.187.254:43017)
0319 16:38:39.800106 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.800108 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.800237 (+ 129us) server_negotiation.cc:408] Connection header received
0319 16:38:39.800578 (+ 341us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.800580 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.800594 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.800670 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.801355 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.801843 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.802381 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.802595 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.804302 (+ 1707us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.804306 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.804306 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.804314 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.805814 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.806159 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.806160 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.806160 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.806188 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.806413 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.806414 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.806415 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.806534 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.806605 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.806805 (+ 200us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.806895 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260319 16:38:39.807006 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.799974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43017 (local address 127.0.0.1:37856)
0319 16:38:39.800114 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.800145 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.800475 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.800708 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.800711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.800973 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.801237 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.801240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.801915 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.801916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.802274 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.802276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.802416 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.802884 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.802901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.804207 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.805891 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.805897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.805899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.806089 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.806273 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.806274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.806275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.806345 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.806592 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.806595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.806692 (+ 97us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.806770 (+ 78us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.806872 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260319 16:38:39.807498 17089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.187.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.187.252@KRBTEST.COM" created.
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:38:39.815884 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.807728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43604)
0319 16:38:39.807778 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.807790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.807838 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.808029 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.808031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.808176 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.808373 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.808375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.809223 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.809225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.809812 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.809815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.809896 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.810303 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.810307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.813360 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.815068 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.815069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.815070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.815195 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.815341 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.815342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.815342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.815369 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.815581 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.815581 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.815621 (+ 40us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.815682 (+ 61us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.815782 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260319 16:38:39.815953 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.807781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.24.187.253:42427)
0319 16:38:39.807917 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.807922 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.807934 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:38:39.807952 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.807953 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.807963 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.808017 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.808458 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.809152 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.809921 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.810104 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.813448 (+ 3344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.813457 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.813457 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.813463 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.814987 (+ 1524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.815263 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.815266 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.815266 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.815293 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.815408 (+ 115us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.815409 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.815409 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.815512 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.815585 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.815716 (+ 131us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.815845 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260319 16:38:39.816491 17089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.816840 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.816732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56450)
0319 16:38:39.816775 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.816819 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260319 16:38:39.816943 17039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
I20260319 16:38:39.817302 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.817228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56452)
0319 16:38:39.817262 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.817284 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260319 16:38:39.817395 17089 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:45249: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.24.187.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.187.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.24.187.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab.
I20260319 16:38:39.849145 25327 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskq2KZu_/build/release/bin/kudu
/tmp/dist-test-taskq2KZu_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:45249
--webserver_interface=127.24.187.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.24.187.212:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.187.254:43017,127.24.187.253:42427,127.24.187.252:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260319 16:38:39.849129 17012 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } attempt: 3
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:38:39.855517 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.849498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.24.187.253:42427)
0319 16:38:39.849554 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.849556 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.849567 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:38:39.849596 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.849598 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.849609 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.849684 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.850124 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.850750 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.851487 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.851656 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.853308 (+ 1652us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.853312 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.853312 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.853319 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.854618 (+ 1299us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.854935 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.854936 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.854936 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.854953 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.855079 (+ 126us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.855080 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.855080 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.855155 (+ 75us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.855197 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.855381 (+ 184us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.855452 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:38:39.855526 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.849420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43606)
0319 16:38:39.849467 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.849480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.849528 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.849689 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.849689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.849847 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.850021 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.850023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.850842 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.850844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.851374 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.851378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.851453 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.851910 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.851915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.853229 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.854695 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.854697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.854697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.854869 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.855002 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.855003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.855003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.855033 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.855203 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.855204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.855255 (+ 51us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.855335 (+ 80us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.855428 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:38:39.855913 17012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.856240 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.856132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56464)
0319 16:38:39.856177 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.856212 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260319 16:38:39.856565 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.856475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56466)
0319 16:38:39.856515 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.856544 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260319 16:38:39.856649 17012 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:45249: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
I20260319 16:38:39.866588 17089 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } attempt: 1
I20260319 16:38:39.867027 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.866891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56472)
0319 16:38:39.866961 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.866998 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260319 16:38:39.867386 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.867282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56486)
0319 16:38:39.867341 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.867366 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260319 16:38:39.867475 17089 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:45249: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
I20260319 16:38:39.887796 17012 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } attempt: 1
I20260319 16:38:39.888281 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.888146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56500)
0319 16:38:39.888205 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.888248 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260319 16:38:39.888618 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.888536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56504)
0319 16:38:39.888574 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.888599 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260319 16:38:39.888700 17012 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:45249: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
W20260319 16:38:39.924621 17098 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260319 16:38:39.924787 17098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260319 16:38:39.924801 17098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260319 16:38:39.924819 17098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260319 16:38:39.926323 17098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260319 16:38:39.926362 17098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260319 16:38:39.926375 17098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260319 16:38:39.926385 17098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260319 16:38:39.926396 17098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260319 16:38:39.927968 17098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.187.212:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.187.254:43017,127.24.187.253:42427,127.24.187.252:45249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/krb5kdc/kudu.keytab
--principal=kudu/127.24.187.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.187.252:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.187.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type RELEASE
built by None at 19 Mar 2026 15:59:19 UTC on bdcb31816ec0
build id 11005
I20260319 16:38:39.928148 17098 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260319 16:38:39.928308 17098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260319 16:38:39.930347 17103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:39.930310 17106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260319 16:38:39.930353 17104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260319 16:38:39.930547 17098 server_base.cc:1060] running on GCE node
I20260319 16:38:39.930718 17098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260319 16:38:39.930920 17098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260319 16:38:39.932045 17098 hybrid_clock.cc:648] HybridClock initialized: now 1773938319932027 us; error 25 us; skew 500 ppm
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260319 16:38:39.934258 17098 init.cc:377] Logged in from keytab as kudu/127.24.187.252@KRBTEST.COM (short username kudu)
I20260319 16:38:39.934744 17098 webserver.cc:492] Webserver started at http://127.24.187.252:40119/ using document root <none> and password file <none>
I20260319 16:38:39.934916 17098 fs_manager.cc:362] Metadata directory not provided
I20260319 16:38:39.934950 17098 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260319 16:38:39.935043 17098 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260319 16:38:39.935612 17089 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } attempt: 2
I20260319 16:38:39.935973 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.935852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56520)
0319 16:38:39.935909 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.935943 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260319 16:38:39.935987 17098 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data/instance:
uuid: "c146880ebd674ffda12dec03844a0f22"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "d1342b8d4085726278d10504dbd58390"
server_key_iv: "12886f83e10fda3049656aadedd66eeb"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.936305 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.936224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56532)
0319 16:38:39.936263 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.936287 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260319 16:38:39.936329 17098 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/wal/instance:
uuid: "c146880ebd674ffda12dec03844a0f22"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "d1342b8d4085726278d10504dbd58390"
server_key_iv: "12886f83e10fda3049656aadedd66eeb"
server_key_version: "encryptionkey@0"
W20260319 16:38:39.936396 17089 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.187.252:45249: Network error: Client connection negotiation failed: client connection to 127.24.187.252:45249: connect: Connection refused (error 111)
I20260319 16:38:39.937389 17098 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260319 16:38:39.938071 17113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:39.938230 17098 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260319 16:38:39.938293 17098 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data,/tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/wal
uuid: "c146880ebd674ffda12dec03844a0f22"
format_stamp: "Formatted at 2026-03-19 16:38:39 on dist-test-slave-svn8"
server_key: "d1342b8d4085726278d10504dbd58390"
server_key_iv: "12886f83e10fda3049656aadedd66eeb"
server_key_version: "encryptionkey@0"
I20260319 16:38:39.938349 17098 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260319 16:38:39.944533 17098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260319 16:38:39.946177 17098 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260319 16:38:39.946275 17098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260319 16:38:39.949747 17098 rpc_server.cc:307] RPC server started. Bound to: 127.24.187.252:45249
I20260319 16:38:39.949796 17165 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.187.252:45249 every 8 connection(s)
I20260319 16:38:39.950063 17098 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/data/info.pb
I20260319 16:38:39.950618 17166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260319 16:38:39.952517 17166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.953585 25327 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskq2KZu_/build/release/bin/kudu as pid 17098
I20260319 16:38:39.953668 25327 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskq2KZu_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1773937967898507-25327-0/minicluster-data/master-2/wal/instance
I20260319 16:38:39.953830 25327 external_mini_cluster.cc:1468] Setting key fb1e01a76aaf584852fb2f2ef1ffa9ba
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:38:39.961042 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.953148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.24.187.254:43017)
0319 16:38:39.953205 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.953207 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.953406 (+ 199us) server_negotiation.cc:408] Connection header received
0319 16:38:39.953819 (+ 413us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.953821 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.953834 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.953899 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.954738 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.955181 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.955723 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.955890 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.957954 (+ 2064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.957958 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.957958 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.957965 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.959553 (+ 1588us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.960142 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.960143 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.960144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.960168 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.960400 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.960400 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.960401 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.960485 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.960533 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.960865 (+ 332us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.960953 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:38:39.961114 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.953112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.254:43017 (local address 127.0.0.1:37872)
0319 16:38:39.953299 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.953325 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.953701 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.953954 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.953958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.954334 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.954636 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.954640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.955274 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.955275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.955647 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.955648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.955762 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.956266 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.956282 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.957871 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.959733 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.959738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.959740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.960050 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.960235 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.960236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.960236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.960342 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.960572 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.960576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.960712 (+ 136us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.960852 (+ 140us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.960973 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
I20260319 16:38:39.961673 17166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.962098 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.954179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.24.187.252:45249)
0319 16:38:39.954346 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.954360 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.954401 (+ 41us) server_negotiation.cc:408] Connection header received
0319 16:38:39.954471 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.954474 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.954490 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.954572 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.955233 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.955934 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.956454 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.956651 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.958587 (+ 1936us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.958594 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.958595 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.958600 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.960873 (+ 2273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.961235 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.961238 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.961238 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.961266 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.961427 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.961428 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.961429 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.961524 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.961684 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.961765 (+ 81us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.962005 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260319 16:38:39.963830 17012 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } attempt: 2
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.254@KRBTEST.COM
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.254@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:38:39.968602 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.962072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.24.187.253:42427)
0319 16:38:39.962121 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.962122 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.962130 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:38:39.962154 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.962155 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.962165 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.962221 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.962611 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.963138 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.963723 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.963867 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.965620 (+ 1753us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.965625 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.965626 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.965633 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.967592 (+ 1959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.967925 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.967926 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.967926 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.967952 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.968097 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.968098 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.968099 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.968208 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.968266 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.968455 (+ 189us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.968524 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:38:39.968892 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.961943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.253:42427 (local address 127.0.0.1:43622)
0319 16:38:39.961977 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.961987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.962021 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.962248 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.962249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.962378 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.962536 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.962539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.963202 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.963203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.963643 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.963645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.963707 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.964086 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.964090 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.965501 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.967678 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.967679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.967680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.967846 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.968013 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.968014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.968014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.968043 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.968289 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.968290 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.968343 (+ 53us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.968681 (+ 338us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.968804 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260319 16:38:39.969465 17166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
I20260319 16:38:39.969766 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.962797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.24.187.254:43017)
0319 16:38:39.962832 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.962834 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.962840 (+ 6us) server_negotiation.cc:408] Connection header received
0319 16:38:39.962858 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.962859 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.962867 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.962917 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.963395 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.964025 (+ 630us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.964766 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.964930 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.966816 (+ 1886us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.966819 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.966820 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.966826 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.968157 (+ 1331us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.968606 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.968607 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.968607 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.968630 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.969132 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.969133 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.969133 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.969233 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.969367 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.969569 (+ 202us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.969655 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:39.971153 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.964045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56550)
0319 16:38:39.964090 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.964099 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.964130 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.964236 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.964237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.964358 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.964493 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.964494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.965890 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.965891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.966290 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.966293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.966328 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.966680 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.966684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.968156 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.970114 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.970115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.970116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.970277 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.970436 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.970438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.970438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.970473 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.970718 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.970719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.970776 (+ 57us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.970957 (+ 181us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.971062 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260319 16:38:39.971470 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.964092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.24.187.252:45249)
0319 16:38:39.964135 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.964136 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.964158 (+ 22us) server_negotiation.cc:408] Connection header received
0319 16:38:39.964173 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.964173 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.964180 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.964221 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.965323 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.965789 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.966490 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.966953 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.968249 (+ 1296us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.968251 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.968252 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.968257 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.970030 (+ 1773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.970362 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.970363 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.970363 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.970384 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.970557 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.970558 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.970558 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.970662 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.970727 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.971320 (+ 593us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.971393 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260319 16:38:39.972714 17012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4: Bootstrap starting.
I20260319 16:38:39.973482 17012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:39.973739 17012 log.cc:826] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4: Log is configured to *not* fsync() on all Append() calls
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.187.253@KRBTEST.COM
I20260319 16:38:39.974677 17012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4: No bootstrap required, opened a new log
Mar 19 16:38:39 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.252@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:38:39.976202 17012 raft_consensus.cc:359] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:39.976317 17012 raft_consensus.cc:740] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58bf5b5ee5274473812dd2d4b926d0d4, State: Initialized, Role: FOLLOWER
I20260319 16:38:39.976409 17012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:39.976732 17012 sys_catalog.cc:565] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:39.976703 17173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:39.976909 17173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:39.977342 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.970516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.24.187.253:42427)
0319 16:38:39.970555 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.970557 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.970564 (+ 7us) server_negotiation.cc:408] Connection header received
0319 16:38:39.970582 (+ 18us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.970583 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.970592 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.970649 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.971299 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.971805 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.972241 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.972396 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.974468 (+ 2072us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.974471 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.974472 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.974477 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.975872 (+ 1395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.976193 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.976194 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.976194 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.976218 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.976402 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.976403 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.976404 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.976505 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.976636 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.977158 (+ 522us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.977251 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:38:39.978935 25327 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20260319 16:38:39.979254 17184 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:39.979296 17184 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:39.979617 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.969741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56552)
0319 16:38:39.970061 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:39.970070 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:39.970106 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:39.970318 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:39.970319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:39.970612 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:39.970767 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.970769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.974100 (+ 3331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.974101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:39.974496 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:39.974498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.974543 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.974997 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:39.975001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:39.976323 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:39.978857 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.978858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.978859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.979014 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.979146 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:39.979146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:39.979146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:39.979168 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:39.979378 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:39.979378 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:39.979408 (+ 30us) client_negotiation.cc:769] Sending connection context
0319 16:38:39.979446 (+ 38us) client_negotiation.cc:240] Negotiation successful
0319 16:38:39.979530 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260319 16:38:39.979905 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.969802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.24.187.252:45249)
0319 16:38:39.970198 (+ 396us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.970199 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.970209 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:38:39.970236 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.970237 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.970247 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.970301 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.970883 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.974004 (+ 3121us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.974775 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.975477 (+ 702us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.976393 (+ 916us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.976396 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.976396 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.976401 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.978796 (+ 2395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.979080 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.979080 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.979081 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.979100 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.979213 (+ 113us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.979214 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.979214 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.979324 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.979713 (+ 389us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.979748 (+ 35us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.979824 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":103,"threads_started":1}
I20260319 16:38:39.980926 17166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: Bootstrap starting.
I20260319 16:38:39.979408 16944 master_runner.cc:186] Error getting master registration for 127.24.187.253:42427: OK, instance_id { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" instance_seqno: 1773938319795584 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260319 16:38:39.980988 16944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260319 16:38:39.981177 16986 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "58bf5b5ee5274473812dd2d4b926d0d4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37880
I20260319 16:38:39.981303 16986 raft_consensus.cc:493] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260319 16:38:39.981351 16986 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:39.981549 17166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:39.981886 17166 log.cc:826] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:39.982270 16986 raft_consensus.cc:515] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:39.982491 16986 leader_election.cc:290] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [CANDIDATE]: Term 1 election: Requested vote from peers 4400ceb324334dbe8794262dbb34fec1 (127.24.187.253:42427), c146880ebd674ffda12dec03844a0f22 (127.24.187.252:45249)
I20260319 16:38:39.982641 17166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: No bootstrap required, opened a new log
I20260319 16:38:39.984138 17166 raft_consensus.cc:359] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:39.984268 17166 raft_consensus.cc:740] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c146880ebd674ffda12dec03844a0f22, State: Initialized, Role: FOLLOWER
I20260319 16:38:39.984356 17166 consensus_queue.cc:260] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:39.984658 17166 sys_catalog.cc:565] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:39.984732 17140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c146880ebd674ffda12dec03844a0f22"
I20260319 16:38:39.984812 17140 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:39.985646 17140 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58bf5b5ee5274473812dd2d4b926d0d4 in term 1.
I20260319 16:38:39.985867 16962 leader_election.cc:304] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58bf5b5ee5274473812dd2d4b926d0d4, c146880ebd674ffda12dec03844a0f22; no voters:
I20260319 16:38:39.985987 17173 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260319 16:38:39.986097 17173 raft_consensus.cc:697] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [term 1 LEADER]: Becoming Leader. State: Replica: 58bf5b5ee5274473812dd2d4b926d0d4, State: Running, Role: LEADER
I20260319 16:38:39.986173 17173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:39.986593 17206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58bf5b5ee5274473812dd2d4b926d0d4. Latest consensus state: current_term: 1 leader_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:39.986979 17206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:39.987455 17208 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260319 16:38:39.987686 17208 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260319 16:38:39.988152 17098 master_runner.cc:186] Error getting master registration for 127.24.187.253:42427: OK, instance_id { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" instance_seqno: 1773938319795584 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260319 16:38:39.989931 17098 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260319 16:38:39.989624 17193 sys_catalog.cc:455] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:39.990921 17193 sys_catalog.cc:458] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:39.993769 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.984124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.24.187.252:45249)
0319 16:38:39.984168 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.984170 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.985279 (+ 1109us) server_negotiation.cc:408] Connection header received
0319 16:38:39.985428 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.985430 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.985441 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.985492 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.986677 (+ 1185us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.987369 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.988104 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.988286 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.990545 (+ 2259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.990548 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.990549 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.990556 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.992326 (+ 1770us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.992677 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.992678 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.992679 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.992705 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.992852 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.992852 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.992853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.992952 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.993064 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.993558 (+ 494us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.993657 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260319 16:38:39.996315 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.983780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.24.187.254:43017)
0319 16:38:39.983831 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.983833 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.986798 (+ 2965us) server_negotiation.cc:408] Connection header received
0319 16:38:39.987279 (+ 481us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.987280 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.987299 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.987351 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.989350 (+ 1999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.990338 (+ 988us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.992420 (+ 2082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.992646 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.994194 (+ 1548us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.994197 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.994198 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.994205 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.995437 (+ 1232us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.995749 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.995750 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.995750 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.995773 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.995896 (+ 123us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.995897 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.995897 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.995982 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.996073 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.996161 (+ 88us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.996226 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260319 16:38:39.997402 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:39.984600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.24.187.253:42427)
0319 16:38:39.984666 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:39.984668 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:39.989658 (+ 4990us) server_negotiation.cc:408] Connection header received
0319 16:38:39.989682 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:39.989683 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:39.989693 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:39.989786 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:39.991480 (+ 1694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.992103 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:39.992782 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:39.992944 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:39.994626 (+ 1682us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:39.994630 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:39.994631 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:39.994637 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:39.996257 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.996749 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.996750 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.996751 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.996775 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:39.996908 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:39.996909 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:39.996909 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:39.997021 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:39.997105 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:39.997234 (+ 129us) server_negotiation.cc:300] Negotiation successful
0319 16:38:39.997312 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260319 16:38:39.998251 17205 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260319 16:38:39.998306 17205 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260319 16:38:39.999363 17140 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [term 1 FOLLOWER]: Refusing update from remote peer 58bf5b5ee5274473812dd2d4b926d0d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:38:39.999676 17206 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:38:40.001103 17193 sys_catalog.cc:455] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58bf5b5ee5274473812dd2d4b926d0d4. Latest consensus state: current_term: 1 leader_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:40.001207 17193 sys_catalog.cc:458] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:40.002681 17173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:40.002760 17173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:40.002885 17209 mvcc.cc:204] Tried to move back new op lower bound from 7266051358714400768 to 7266051358663983104. Current Snapshot: MvccSnapshot[applied={T|T < 7266051358714400768}]
I20260319 16:38:40.003180 17193 sys_catalog.cc:455] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:40.003268 17193 sys_catalog.cc:458] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:40.004606 17211 mvcc.cc:204] Tried to move back new op lower bound from 7266051358714400768 to 7266051358663983104. Current Snapshot: MvccSnapshot[applied={T|T < 7266051358714400768 or (T in {7266051358714400768})}]
I20260319 16:38:40.010255 17208 catalog_manager.cc:1357] Generated new cluster ID: 6e5c7631247842e29bdd36065e37092b
I20260319 16:38:40.010315 17208 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260319 16:38:40.026662 17208 catalog_manager.cc:1380] Generated new certificate authority record
I20260319 16:38:40.027451 17208 catalog_manager.cc:1514] Loading token signing keys...
I20260319 16:38:40.027514 17089 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } attempt: 3
Mar 19 16:38:40 dist-test-slave-svn8 krb5kdc[16928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1773938319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.187.253@KRBTEST.COM for kudu/127.24.187.252@KRBTEST.COM
I20260319 16:38:40.035917 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.027756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.187.252:45249 (local address 127.0.0.1:56572)
0319 16:38:40.027855 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0319 16:38:40.027867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0319 16:38:40.027904 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0319 16:38:40.028077 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0319 16:38:40.028079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0319 16:38:40.028273 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0319 16:38:40.028470 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:40.028472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.029456 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.029457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0319 16:38:40.030051 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0319 16:38:40.030053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.030117 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.031311 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0319 16:38:40.031316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0319 16:38:40.032944 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0319 16:38:40.034866 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:40.034868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:40.034868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:40.035059 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:40.035259 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0319 16:38:40.035260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0319 16:38:40.035260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0319 16:38:40.035304 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0319 16:38:40.035601 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0319 16:38:40.035602 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0319 16:38:40.035661 (+ 59us) client_negotiation.cc:769] Sending connection context
0319 16:38:40.035719 (+ 58us) client_negotiation.cc:240] Negotiation successful
0319 16:38:40.035816 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260319 16:38:40.036486 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.027872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.24.187.252:45249)
0319 16:38:40.027911 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.027913 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.027926 (+ 13us) server_negotiation.cc:408] Connection header received
0319 16:38:40.027950 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.027951 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.027962 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.028018 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.028566 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.029348 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.031042 (+ 1694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.031223 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.033038 (+ 1815us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0319 16:38:40.033041 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0319 16:38:40.033042 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0319 16:38:40.033049 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0319 16:38:40.034760 (+ 1711us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:40.035145 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:40.035146 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:40.035147 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:40.035186 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0319 16:38:40.035375 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0319 16:38:40.035376 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0319 16:38:40.035376 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0319 16:38:40.035497 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0319 16:38:40.035562 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.036308 (+ 746us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.036391 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260319 16:38:40.038040 17089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1: Bootstrap starting.
I20260319 16:38:40.038925 17089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1: Neither blocks nor log segments found. Creating new log.
I20260319 16:38:40.039171 17089 log.cc:826] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1: Log is configured to *not* fsync() on all Append() calls
I20260319 16:38:40.040096 17089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1: No bootstrap required, opened a new log
I20260319 16:38:40.043310 17208 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4: Generated new TSK 0
I20260319 16:38:40.043493 17208 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260319 16:38:40.044083 17089 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:40.044387 17089 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4400ceb324334dbe8794262dbb34fec1, State: Initialized, Role: FOLLOWER
I20260319 16:38:40.044575 17089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } }
I20260319 16:38:40.045056 17217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:40.045116 17089 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260319 16:38:40.045140 17217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:40.045277 17062 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [term 0 FOLLOWER]: Advancing to term 1
I20260319 16:38:40.046438 17062 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [term 1 FOLLOWER]: Refusing update from remote peer 58bf5b5ee5274473812dd2d4b926d0d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260319 16:38:40.045167 17063 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4400ceb324334dbe8794262dbb34fec1"
I20260319 16:38:40.046729 17214 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260319 16:38:40.047499 17063 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58bf5b5ee5274473812dd2d4b926d0d4 in term 1.
I20260319 16:38:40.049031 17217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58bf5b5ee5274473812dd2d4b926d0d4. Latest consensus state: current_term: 1 leader_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:40.049105 17217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:40.049230 17220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:40.049418 17220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1 [sys.catalog]: This master's current role is: FOLLOWER
I20260319 16:38:40.051786 17234 catalog_manager.cc:1269] Loaded cluster ID: 6e5c7631247842e29bdd36065e37092b
I20260319 16:38:40.051831 17234 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1: loading cluster ID for follower catalog manager: success
I20260319 16:38:40.053124 17234 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1: acquiring CA information for follower catalog manager: success
I20260319 16:38:40.053423 17234 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 4400ceb324334dbe8794262dbb34fec1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260319 16:38:40.068593 17015 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.067306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.24.187.254:43017)
0319 16:38:40.067351 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.067353 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.067937 (+ 584us) server_negotiation.cc:408] Connection header received
0319 16:38:40.068039 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.068041 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.068052 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.068082 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.068515 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37900: BlockingRecv error: recv got EOF from 127.0.0.1:37900 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260319 16:38:40.069770 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.067663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.24.187.252:45249)
0319 16:38:40.067701 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.067703 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.069231 (+ 1528us) server_negotiation.cc:408] Connection header received
0319 16:38:40.069325 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.069326 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.069336 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.069363 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.069694 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56574: BlockingRecv error: recv got EOF from 127.0.0.1:56574 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260319 16:38:40.070729 17093 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.069114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.24.187.253:42427)
0319 16:38:40.069152 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.069154 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.070136 (+ 982us) server_negotiation.cc:408] Connection header received
0319 16:38:40.070301 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.070302 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.070312 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.070340 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.070658 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43664: BlockingRecv error: recv got EOF from 127.0.0.1:43664 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:38:40.080956 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.077865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.24.187.253:42427)
0319 16:38:40.077905 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.077906 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.078037 (+ 131us) server_negotiation.cc:408] Connection header received
0319 16:38:40.078152 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.078153 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.078179 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.078212 (+ 33us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.078647 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.079406 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.080264 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.080439 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.080583 (+ 144us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.080701 (+ 118us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.080749 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.080852 (+ 103us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.080877 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260319 16:38:40.084501 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.083762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.24.187.252:45249)
0319 16:38:40.083808 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.083810 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.083937 (+ 127us) server_negotiation.cc:408] Connection header received
0319 16:38:40.084041 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.084042 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.084057 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.084086 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.084446 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56590: BlockingRecv error: recv got EOF from 127.0.0.1:56590 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260319 16:38:40.092504 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.086869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.24.187.253:42427)
0319 16:38:40.086906 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.086908 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.089898 (+ 2990us) server_negotiation.cc:408] Connection header received
0319 16:38:40.089997 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.089998 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.090009 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.090037 (+ 28us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.090401 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.091165 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.091972 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.092145 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.092246 (+ 101us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.092275 (+ 29us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.092304 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.092404 (+ 100us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.092426 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260319 16:38:40.096017 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.095203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.24.187.252:45249)
0319 16:38:40.095242 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.095244 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.095459 (+ 215us) server_negotiation.cc:408] Connection header received
0319 16:38:40.095568 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.095569 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.095581 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.095609 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.095962 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56594: BlockingRecv error: recv got EOF from 127.0.0.1:56594 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:40.102139 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.099296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.24.187.253:42427)
0319 16:38:40.099331 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.099333 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.099551 (+ 218us) server_negotiation.cc:408] Connection header received
0319 16:38:40.099642 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.099643 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.099653 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.099680 (+ 27us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.100041 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.100782 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.101587 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.101785 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.101894 (+ 109us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.101923 (+ 29us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.101950 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.102047 (+ 97us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.102067 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260319 16:38:40.105286 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.104498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.24.187.252:45249)
0319 16:38:40.104547 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.104549 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.104770 (+ 221us) server_negotiation.cc:408] Connection header received
0319 16:38:40.104861 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.104862 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.104873 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.104901 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.105234 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56608: BlockingRecv error: recv got EOF from 127.0.0.1:56608 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:40.113427 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.110622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.24.187.253:42427)
0319 16:38:40.110660 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.110662 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.110873 (+ 211us) server_negotiation.cc:408] Connection header received
0319 16:38:40.110965 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.110966 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.110976 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.111003 (+ 27us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.111358 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.112095 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.112896 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.113072 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.113190 (+ 118us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.113219 (+ 29us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.113244 (+ 25us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.113336 (+ 92us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.113355 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260319 16:38:40.116578 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.115838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.24.187.252:45249)
0319 16:38:40.115877 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.115879 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.116060 (+ 181us) server_negotiation.cc:408] Connection header received
0319 16:38:40.116146 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.116147 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.116157 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.116183 (+ 26us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.116515 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56622: BlockingRecv error: recv got EOF from 127.0.0.1:56622 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260319 16:38:40.128969 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.126008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.24.187.253:42427)
0319 16:38:40.126049 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.126051 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.126308 (+ 257us) server_negotiation.cc:408] Connection header received
0319 16:38:40.126411 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.126412 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.126423 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.126451 (+ 28us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.126842 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.127611 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.128409 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.128569 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.128703 (+ 134us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.128733 (+ 30us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.128760 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.128867 (+ 107us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.128892 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260319 16:38:40.132462 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.131624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.24.187.252:45249)
0319 16:38:40.131664 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.131666 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.131900 (+ 234us) server_negotiation.cc:408] Connection header received
0319 16:38:40.131996 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.131997 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.132008 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.132036 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.132403 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56624: BlockingRecv error: recv got EOF from 127.0.0.1:56624 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:38:40.153297 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.150163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.24.187.253:42427)
0319 16:38:40.150246 (+ 83us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.150248 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.150535 (+ 287us) server_negotiation.cc:408] Connection header received
0319 16:38:40.150647 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.150649 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.150661 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.150691 (+ 30us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.151081 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.151842 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.152692 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.152863 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.152985 (+ 122us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.153017 (+ 32us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.153046 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.153178 (+ 132us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.153203 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260319 16:38:40.156800 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.155966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.24.187.252:45249)
0319 16:38:40.156006 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.156009 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.156238 (+ 229us) server_negotiation.cc:408] Connection header received
0319 16:38:40.156346 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.156348 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.156359 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.156386 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.156746 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56638: BlockingRecv error: recv got EOF from 127.0.0.1:56638 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260319 16:38:40.194258 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.191313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.24.187.253:42427)
0319 16:38:40.191357 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.191359 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.191639 (+ 280us) server_negotiation.cc:408] Connection header received
0319 16:38:40.191734 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.191735 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.191746 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.191772 (+ 26us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.192129 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.192885 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.193751 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.193920 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.194015 (+ 95us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.194043 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.194067 (+ 24us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.194156 (+ 89us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.194175 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260319 16:38:40.197698 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.196918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.24.187.252:45249)
0319 16:38:40.196957 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.196959 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.197181 (+ 222us) server_negotiation.cc:408] Connection header received
0319 16:38:40.197272 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.197273 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.197294 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.197320 (+ 26us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.197646 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56644: BlockingRecv error: recv got EOF from 127.0.0.1:56644 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260319 16:38:40.265134 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.263218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.24.187.253:42427)
0319 16:38:40.263269 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.263271 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.263351 (+ 80us) server_negotiation.cc:408] Connection header received
0319 16:38:40.263417 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.263418 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.263430 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.263471 (+ 41us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.263730 (+ 259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.264190 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.264752 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.264880 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.264897 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.264919 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.264963 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.265037 (+ 74us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.265056 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260319 16:38:40.267355 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.266788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.24.187.252:45249)
0319 16:38:40.266849 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.266852 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.266863 (+ 11us) server_negotiation.cc:408] Connection header received
0319 16:38:40.266924 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.266925 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.266935 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.266972 (+ 37us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.267265 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56654: BlockingRecv error: recv got EOF from 127.0.0.1:56654 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260319 16:38:40.398365 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.396461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.24.187.253:42427)
0319 16:38:40.396530 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.396531 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.396543 (+ 12us) server_negotiation.cc:408] Connection header received
0319 16:38:40.396603 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.396605 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.396616 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.396663 (+ 47us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.396901 (+ 238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.397390 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.397972 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.398111 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.398129 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.398150 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.398195 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.398265 (+ 70us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.398285 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260319 16:38:40.400501 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.399957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.24.187.252:45249)
0319 16:38:40.400030 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.400033 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.400042 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:38:40.400101 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.400102 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.400114 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.400161 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.400443 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56662: BlockingRecv error: recv got EOF from 127.0.0.1:56662 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260319 16:38:40.658812 17214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58bf5b5ee5274473812dd2d4b926d0d4" member_type: VOTER last_known_addr { host: "127.24.187.254" port: 43017 } } peers { permanent_uuid: "4400ceb324334dbe8794262dbb34fec1" member_type: VOTER last_known_addr { host: "127.24.187.253" port: 42427 } } peers { permanent_uuid: "c146880ebd674ffda12dec03844a0f22" member_type: VOTER last_known_addr { host: "127.24.187.252" port: 45249 } } }
I20260319 16:38:40.658941 17214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58bf5b5ee5274473812dd2d4b926d0d4 [sys.catalog]: This master's current role is: LEADER
I20260319 16:38:40.659968 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:40.657765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.24.187.253:42427)
0319 16:38:40.657829 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.657831 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.657866 (+ 35us) server_negotiation.cc:408] Connection header received
0319 16:38:40.657979 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.657981 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.657992 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.658051 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:40.658383 (+ 332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.658948 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:40.659613 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:40.659744 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:40.659761 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:40.659783 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:40.659829 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:40.659898 (+ 69us) server_negotiation.cc:300] Negotiation successful
0319 16:38:40.659914 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260319 16:38:40.662055 17171 negotiation.cc:337] Failed RPC negotiation. Trace:
0319 16:38:40.661505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.24.187.252:45249)
0319 16:38:40.661578 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:40.661580 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:40.661588 (+ 8us) server_negotiation.cc:408] Connection header received
0319 16:38:40.661614 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:40.661615 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:40.661626 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:40.661669 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0319 16:38:40.662002 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56664: BlockingRecv error: recv got EOF from 127.0.0.1:56664 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260319 16:38:40.998701 17205 catalog_manager.cc:1269] Loaded cluster ID: 6e5c7631247842e29bdd36065e37092b
I20260319 16:38:40.998768 17205 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: loading cluster ID for follower catalog manager: success
I20260319 16:38:40.999622 17205 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: acquiring CA information for follower catalog manager: success
I20260319 16:38:40.999866 17205 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c146880ebd674ffda12dec03844a0f22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260319 16:38:41.177186 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:41.175209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.24.187.253:42427)
0319 16:38:41.175382 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:41.175384 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:41.175393 (+ 9us) server_negotiation.cc:408] Connection header received
0319 16:38:41.175421 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:41.175422 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:41.175431 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:41.175484 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:41.175730 (+ 246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.176212 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:41.176776 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.176918 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:41.176936 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:41.176960 (+ 24us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:41.177006 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:41.177071 (+ 65us) server_negotiation.cc:300] Negotiation successful
0319 16:38:41.177094 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260319 16:38:41.181325 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:41.179152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.24.187.252:45249)
0319 16:38:41.179315 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:41.179318 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:41.179328 (+ 10us) server_negotiation.cc:408] Connection header received
0319 16:38:41.179375 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:41.179376 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:41.179413 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:41.179476 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:41.179723 (+ 247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.180204 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:41.180776 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.180891 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:41.180929 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:41.181020 (+ 91us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:41.181081 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:41.181206 (+ 125us) server_negotiation.cc:300] Negotiation successful
0319 16:38:41.181239 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260319 16:38:41.185653 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:41.183154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.24.187.254:43017)
0319 16:38:41.183269 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:41.183271 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:41.183344 (+ 73us) server_negotiation.cc:408] Connection header received
0319 16:38:41.183472 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:41.183474 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:41.183507 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:41.183570 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:41.183798 (+ 228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.184494 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:41.185120 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.185274 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:41.185341 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:41.185444 (+ 103us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:41.185526 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:41.185544 (+ 18us) server_negotiation.cc:300] Negotiation successful
0319 16:38:41.185571 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260319 16:38:41.185842 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:41.183287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.24.187.252:45249)
0319 16:38:41.183336 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:41.183337 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:41.183528 (+ 191us) server_negotiation.cc:408] Connection header received
0319 16:38:41.183622 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:41.183623 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:41.183635 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:41.183663 (+ 28us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:41.183990 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.184575 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:41.185341 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.185518 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:41.185545 (+ 27us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:41.185573 (+ 28us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:41.185629 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:41.185748 (+ 119us) server_negotiation.cc:300] Negotiation successful
0319 16:38:41.185776 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260319 16:38:41.186308 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0319 16:38:41.183190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.24.187.253:42427)
0319 16:38:41.183245 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0319 16:38:41.183247 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0319 16:38:41.183444 (+ 197us) server_negotiation.cc:408] Connection header received
0319 16:38:41.183556 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0319 16:38:41.183557 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0319 16:38:41.183571 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0319 16:38:41.183619 (+ 48us) server_negotiation.cc:227] Negotiated authn=TOKEN
0319 16:38:41.183944 (+ 325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.184632 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0319 16:38:41.185664 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0319 16:38:41.185867 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0319 16:38:41.185935 (+ 68us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0319 16:38:41.185967 (+ 32us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0319 16:38:41.186047 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0319 16:38:41.186119 (+ 72us) server_negotiation.cc:300] Negotiation successful
0319 16:38:41.186148 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260319 16:38:41.187786 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 16944
I20260319 16:38:41.194748 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 17021
I20260319 16:38:41.199303 25327 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskq2KZu_/build/release/bin/kudu with pid 17098
2026-03-19T16:38:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3662 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7264 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (353314 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST